2015-03-13 08:41:41 -04:00
|
|
|
|
PEP: 489
|
2015-05-18 10:27:02 -04:00
|
|
|
|
Title: Multi-phase extension module initialization
|
2015-03-13 08:41:41 -04:00
|
|
|
|
Version: $Revision$
|
|
|
|
|
Last-Modified: $Date$
|
|
|
|
|
Author: Petr Viktorin <encukou@gmail.com>,
|
|
|
|
|
Stefan Behnel <stefan_ml@behnel.de>,
|
|
|
|
|
Nick Coghlan <ncoghlan@gmail.com>
|
2015-05-20 17:27:54 -04:00
|
|
|
|
BDFL-Delegate: Eric Snow <ericsnowcurrently@gmail.com>
|
2015-03-13 08:41:41 -04:00
|
|
|
|
Discussions-To: import-sig@python.org
|
2015-05-22 17:45:38 -04:00
|
|
|
|
Status: Final
|
2015-03-13 08:41:41 -04:00
|
|
|
|
Type: Standards Track
|
|
|
|
|
Content-Type: text/x-rst
|
|
|
|
|
Created: 11-Aug-2013
|
|
|
|
|
Python-Version: 3.5
|
2022-03-09 11:04:44 -05:00
|
|
|
|
Post-History: 23-Aug-2013, 20-Feb-2015, 16-Apr-2015, 07-May-2015, 18-May-2015
|
2015-05-22 17:45:38 -04:00
|
|
|
|
Resolution: https://mail.python.org/pipermail/python-dev/2015-May/140108.html
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Abstract
|
|
|
|
|
========
|
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
This PEP proposes a redesign of the way in which built-in and extension modules
|
|
|
|
|
interact with the import machinery. This was last revised for Python 3.0 in PEP
|
2015-05-20 07:27:55 -04:00
|
|
|
|
3121, but did not solve all problems at the time. The goal is to solve
|
|
|
|
|
import-related problems by bringing extension modules closer to the way Python
|
|
|
|
|
modules behave; specifically to hook into the ModuleSpec-based loading
|
2022-01-21 06:03:51 -05:00
|
|
|
|
mechanism introduced in :pep:`451`.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2022-01-21 06:03:51 -05:00
|
|
|
|
This proposal draws inspiration from PyType_Spec of :pep:`384` to allow extension
|
2015-04-16 11:49:43 -04:00
|
|
|
|
authors to only define features they need, and to allow future additions
|
|
|
|
|
to extension module declarations.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
Extensions modules are created in a two-step process, fitting better into
|
|
|
|
|
the ModuleSpec architecture, with parallels to __new__ and __init__ of classes.
|
|
|
|
|
|
|
|
|
|
Extension modules can safely store arbitrary C-level per-module state in
|
|
|
|
|
the module that is covered by normal garbage collection and supports
|
|
|
|
|
reloading and sub-interpreters.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
Extension authors are encouraged to take these issues into account
|
|
|
|
|
when using the new API.
|
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
The proposal also allows extension modules with non-ASCII names.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2022-01-21 06:03:51 -05:00
|
|
|
|
Not all problems tackled in :pep:`3121` are solved in this proposal.
|
2015-05-20 07:27:55 -04:00
|
|
|
|
In particular, problems with run-time module lookup (PyState_FindModule)
|
|
|
|
|
are left to a future PEP.
|
|
|
|
|
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
|
|
|
|
Motivation
|
|
|
|
|
==========
|
|
|
|
|
|
|
|
|
|
Python modules and extension modules are not being set up in the same way.
|
2015-05-18 10:27:02 -04:00
|
|
|
|
For Python modules, the module object is created and set up first, then the
|
2022-01-21 06:03:51 -05:00
|
|
|
|
module code is being executed (:pep:`302`).
|
|
|
|
|
A ModuleSpec object (:pep:`451`) is used to hold information about the module,
|
2015-03-13 08:41:41 -04:00
|
|
|
|
and passed to the relevant hooks.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
For extensions (i.e. shared libraries) and built-in modules, the module
|
2015-03-13 08:41:41 -04:00
|
|
|
|
init function is executed straight away and does both the creation and
|
2015-04-16 11:49:43 -04:00
|
|
|
|
initialization. The initialization function is not passed the ModuleSpec,
|
|
|
|
|
or any information it contains, such as the __file__ or fully-qualified
|
2015-03-13 08:41:41 -04:00
|
|
|
|
name. This hinders relative imports and resource loading.
|
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
In Py3, modules are also not being added to sys.modules, which means that a
|
|
|
|
|
(potentially transitive) re-import of the module will really try to re-import
|
|
|
|
|
it and thus run into an infinite loop when it executes the module init function
|
2015-05-18 10:27:02 -04:00
|
|
|
|
again. Without access to the fully-qualified module name, it is not trivial to
|
|
|
|
|
correctly add the module to sys.modules either.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
This is specifically a problem for Cython generated modules, for which it's
|
|
|
|
|
not uncommon that the module init code has the same level of complexity as
|
|
|
|
|
that of any 'regular' Python module. Also, the lack of __file__ and __name__
|
2015-04-16 11:49:43 -04:00
|
|
|
|
information hinders the compilation of "__init__.py" modules, i.e. packages,
|
2015-03-13 08:41:41 -04:00
|
|
|
|
especially when relative imports are being used at module init time.
|
|
|
|
|
|
|
|
|
|
Furthermore, the majority of currently existing extension modules has
|
2015-04-16 11:49:43 -04:00
|
|
|
|
problems with sub-interpreter support and/or interpreter reloading, and, while
|
|
|
|
|
it is possible with the current infrastructure to support these
|
2015-03-13 08:41:41 -04:00
|
|
|
|
features, it is neither easy nor efficient.
|
2022-01-21 06:03:51 -05:00
|
|
|
|
Addressing these issues was the goal of :pep:`3121`, but many extensions,
|
2015-03-13 08:41:41 -04:00
|
|
|
|
including some in the standard library, took the least-effort approach
|
|
|
|
|
to porting to Python 3, leaving these issues unresolved.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
This PEP keeps backwards compatibility, which should reduce pressure and give
|
|
|
|
|
extension authors adequate time to consider these issues when porting.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The current process
|
|
|
|
|
===================
|
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
Currently, extension and built-in modules export an initialization function
|
|
|
|
|
named "PyInit_modulename", named after the file name of the shared library.
|
|
|
|
|
This function is executed by the import machinery and must return a fully
|
|
|
|
|
initialized module object.
|
|
|
|
|
The function receives no arguments, so it has no way of knowing about its
|
2015-03-13 08:41:41 -04:00
|
|
|
|
import context.
|
|
|
|
|
|
|
|
|
|
During its execution, the module init function creates a module object
|
2015-05-18 10:27:02 -04:00
|
|
|
|
based on a PyModuleDef object. It then continues to initialize it by adding
|
2015-03-13 08:41:41 -04:00
|
|
|
|
attributes to the module dict, creating types, etc.
|
|
|
|
|
|
|
|
|
|
In the back, the shared library loader keeps a note of the fully qualified
|
|
|
|
|
module name of the last module that it loaded, and when a module gets
|
|
|
|
|
created that has a matching name, this global variable is used to determine
|
|
|
|
|
the fully qualified name of the module object. This is not entirely safe as it
|
|
|
|
|
relies on the module init function creating its own module object first,
|
|
|
|
|
but this assumption usually holds in practice.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The proposal
|
|
|
|
|
============
|
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
The initialization function (PyInit_modulename) will be allowed to return
|
|
|
|
|
a pointer to a PyModuleDef object. The import machinery will be in charge
|
|
|
|
|
of constructing the module object, calling hooks provided in the PyModuleDef
|
|
|
|
|
in the relevant phases of initialization (as described below).
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
This multi-phase initialization is an additional possibility. Single-phase
|
|
|
|
|
initialization, the current practice of returning a fully initialized module
|
|
|
|
|
object, will still be accepted, so existing code will work unchanged,
|
|
|
|
|
including binary compatibility.
|
2015-05-07 21:58:19 -04:00
|
|
|
|
|
|
|
|
|
The PyModuleDef structure will be changed to contain a list of slots,
|
2022-01-21 06:03:51 -05:00
|
|
|
|
similarly to :pep:`384`'s PyType_Spec for types.
|
2015-05-07 21:58:19 -04:00
|
|
|
|
To keep binary compatibility, and avoid needing to introduce a new structure
|
|
|
|
|
(which would introduce additional supporting functions and per-module storage),
|
|
|
|
|
the currently unused m_reload pointer of PyModuleDef will be changed to
|
|
|
|
|
hold the slots. The structures are defined as::
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
typedef struct {
|
|
|
|
|
int slot;
|
|
|
|
|
void *value;
|
2015-05-07 21:58:19 -04:00
|
|
|
|
} PyModuleDef_Slot;
|
|
|
|
|
|
|
|
|
|
typedef struct PyModuleDef {
|
|
|
|
|
PyModuleDef_Base m_base;
|
|
|
|
|
const char* m_name;
|
|
|
|
|
const char* m_doc;
|
|
|
|
|
Py_ssize_t m_size;
|
|
|
|
|
PyMethodDef *m_methods;
|
|
|
|
|
PyModuleDef_Slot *m_slots; /* changed from `inquiry m_reload;` */
|
|
|
|
|
traverseproc m_traverse;
|
|
|
|
|
inquiry m_clear;
|
|
|
|
|
freefunc m_free;
|
|
|
|
|
} PyModuleDef;
|
|
|
|
|
|
|
|
|
|
The *m_slots* member must be either NULL, or point to an array of
|
|
|
|
|
PyModuleDef_Slot structures, terminated by a slot with id set to 0
|
|
|
|
|
(i.e. ``{0, NULL}``).
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
To specify a slot, a unique slot ID must be provided.
|
|
|
|
|
New Python versions may introduce new slot IDs, but slot IDs will never be
|
|
|
|
|
recycled. Slots may get deprecated, but will continue to be supported
|
|
|
|
|
throughout Python 3.x.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
A slot's value pointer may not be NULL, unless specified otherwise in the
|
|
|
|
|
slot's documentation.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
The following slots are currently available, and described later:
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
|
|
|
|
* Py_mod_create
|
|
|
|
|
* Py_mod_exec
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
Unknown slot IDs will cause the import to fail with SystemError.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
When using multi-phase initialization, the *m_name* field of PyModuleDef will
|
|
|
|
|
not be used during importing; the module name will be taken from the ModuleSpec.
|
|
|
|
|
|
2015-05-21 08:48:38 -04:00
|
|
|
|
Before it is returned from PyInit_*, the PyModuleDef object must be initialized
|
|
|
|
|
using the newly added PyModuleDef_Init function. This sets the object type
|
|
|
|
|
(which cannot be done statically on certain compilers), refcount, and internal
|
|
|
|
|
bookkeeping data (m_index).
|
2015-05-18 10:27:02 -04:00
|
|
|
|
For example, an extension module "example" would be exported as::
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
static PyModuleDef example_def = {...}
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
PyMODINIT_FUNC
|
|
|
|
|
PyInit_example(void)
|
|
|
|
|
{
|
|
|
|
|
return PyModuleDef_Init(&example_def);
|
|
|
|
|
}
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
The PyModuleDef object must be available for the lifetime of the module created
|
|
|
|
|
from it – usually, it will be declared statically.
|
|
|
|
|
|
2015-05-20 07:27:55 -04:00
|
|
|
|
Pseudo-code Overview
|
|
|
|
|
--------------------
|
|
|
|
|
|
|
|
|
|
Here is an overview of how the modified importers will operate.
|
|
|
|
|
Details such as logging or handling of errors and invalid states
|
|
|
|
|
are left out, and C code is presented with a concise Python-like syntax.
|
|
|
|
|
|
2022-01-21 06:03:51 -05:00
|
|
|
|
The framework that calls the importers is explained in
|
|
|
|
|
:pep:`451#how-loading-will-work`.
|
2015-05-20 07:27:55 -04:00
|
|
|
|
|
2015-05-21 10:10:14 -04:00
|
|
|
|
importlib/_bootstrap.py:
|
2015-05-20 07:27:55 -04:00
|
|
|
|
|
2015-05-21 10:10:14 -04:00
|
|
|
|
::
|
|
|
|
|
|
|
|
|
|
class BuiltinImporter:
|
|
|
|
|
def create_module(self, spec):
|
|
|
|
|
module = _imp.create_builtin(spec)
|
|
|
|
|
|
|
|
|
|
def exec_module(self, module):
|
|
|
|
|
_imp.exec_dynamic(module)
|
|
|
|
|
|
|
|
|
|
def load_module(self, name):
|
|
|
|
|
# use a backwards compatibility shim
|
|
|
|
|
_load_module_shim(self, name)
|
|
|
|
|
|
|
|
|
|
importlib/_bootstrap_external.py:
|
|
|
|
|
|
|
|
|
|
::
|
|
|
|
|
|
|
|
|
|
class ExtensionFileLoader:
|
|
|
|
|
def create_module(self, spec):
|
|
|
|
|
module = _imp.create_dynamic(spec)
|
|
|
|
|
|
|
|
|
|
def exec_module(self, module):
|
|
|
|
|
_imp.exec_dynamic(module)
|
|
|
|
|
|
|
|
|
|
def load_module(self, name):
|
|
|
|
|
# use a backwards compatibility shim
|
|
|
|
|
_load_module_shim(self, name)
|
|
|
|
|
|
|
|
|
|
Python/import.c (the _imp module):
|
|
|
|
|
|
|
|
|
|
::
|
|
|
|
|
|
|
|
|
|
def create_dynamic(spec):
|
|
|
|
|
name = spec.name
|
|
|
|
|
path = spec.origin
|
|
|
|
|
|
|
|
|
|
# Find an already loaded module that used single-phase init.
|
|
|
|
|
# For multi-phase initialization, mod is NULL, so a new module
|
|
|
|
|
# is always created.
|
|
|
|
|
mod = _PyImport_FindExtensionObject(name, name)
|
|
|
|
|
if mod:
|
|
|
|
|
return mod
|
|
|
|
|
|
|
|
|
|
return _PyImport_LoadDynamicModuleWithSpec(spec)
|
|
|
|
|
|
|
|
|
|
def exec_dynamic(module):
|
|
|
|
|
if not isinstance(module, types.ModuleType):
|
|
|
|
|
# non-modules are skipped -- PyModule_GetDef fails on them
|
|
|
|
|
return
|
|
|
|
|
|
|
|
|
|
def = PyModule_GetDef(module)
|
|
|
|
|
state = PyModule_GetState(module)
|
|
|
|
|
if state is NULL:
|
|
|
|
|
PyModule_ExecDef(module, def)
|
|
|
|
|
|
|
|
|
|
def create_builtin(spec):
|
|
|
|
|
name = spec.name
|
|
|
|
|
|
|
|
|
|
# Find an already loaded module that used single-phase init.
|
|
|
|
|
# For multi-phase initialization, mod is NULL, so a new module
|
|
|
|
|
# is always created.
|
|
|
|
|
mod = _PyImport_FindExtensionObject(name, name)
|
|
|
|
|
if mod:
|
|
|
|
|
return mod
|
|
|
|
|
|
|
|
|
|
for initname, initfunc in PyImport_Inittab:
|
|
|
|
|
if name == initname:
|
|
|
|
|
m = initfunc()
|
|
|
|
|
if isinstance(m, PyModuleDef):
|
|
|
|
|
def = m
|
|
|
|
|
return PyModule_FromDefAndSpec(def, spec)
|
|
|
|
|
else:
|
|
|
|
|
# fall back to single-phase initialization
|
|
|
|
|
module = m
|
|
|
|
|
_PyImport_FixupExtensionObject(module, name, name)
|
|
|
|
|
return module
|
|
|
|
|
|
|
|
|
|
Python/importdl.c:
|
|
|
|
|
|
|
|
|
|
::
|
|
|
|
|
|
|
|
|
|
def _PyImport_LoadDynamicModuleWithSpec(spec):
|
|
|
|
|
path = spec.origin
|
|
|
|
|
package, dot, name = spec.name.rpartition('.')
|
|
|
|
|
|
|
|
|
|
# see the "Non-ASCII module names" section for export_hook_name
|
|
|
|
|
hook_name = export_hook_name(name)
|
|
|
|
|
|
|
|
|
|
# call platform-specific function for loading exported function
|
|
|
|
|
# from shared library
|
|
|
|
|
exportfunc = _find_shared_funcptr(hook_name, path)
|
|
|
|
|
|
|
|
|
|
m = exportfunc()
|
|
|
|
|
if isinstance(m, PyModuleDef):
|
|
|
|
|
def = m
|
|
|
|
|
return PyModule_FromDefAndSpec(def, spec)
|
|
|
|
|
|
|
|
|
|
module = m
|
|
|
|
|
|
|
|
|
|
# fall back to single-phase initialization
|
|
|
|
|
....
|
|
|
|
|
|
|
|
|
|
Objects/moduleobject.c:
|
|
|
|
|
|
|
|
|
|
::
|
|
|
|
|
|
|
|
|
|
def PyModule_FromDefAndSpec(def, spec):
|
|
|
|
|
name = spec.name
|
|
|
|
|
create = None
|
|
|
|
|
for slot, value in def.m_slots:
|
|
|
|
|
if slot == Py_mod_create:
|
|
|
|
|
create = value
|
|
|
|
|
if create:
|
|
|
|
|
m = create(spec, def)
|
|
|
|
|
else:
|
|
|
|
|
m = PyModule_New(name)
|
|
|
|
|
|
|
|
|
|
if isinstance(m, types.ModuleType):
|
|
|
|
|
m.md_state = None
|
|
|
|
|
m.md_def = def
|
|
|
|
|
|
|
|
|
|
if def.m_methods:
|
|
|
|
|
PyModule_AddFunctions(m, def.m_methods)
|
|
|
|
|
if def.m_doc:
|
|
|
|
|
PyModule_SetDocString(m, def.m_doc)
|
|
|
|
|
|
|
|
|
|
def PyModule_ExecDef(module, def):
|
|
|
|
|
if isinstance(module, types.module_type):
|
|
|
|
|
if module.md_state is NULL:
|
|
|
|
|
# allocate a block of zeroed-out memory
|
|
|
|
|
module.md_state = _alloc(module.md_size)
|
|
|
|
|
|
|
|
|
|
if def.m_slots is NULL:
|
|
|
|
|
return
|
|
|
|
|
|
|
|
|
|
for slot, value in def.m_slots:
|
|
|
|
|
if slot == Py_mod_exec:
|
|
|
|
|
value(module)
|
2015-05-20 07:27:55 -04:00
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
|
|
|
|
|
Module Creation Phase
|
|
|
|
|
---------------------
|
|
|
|
|
|
|
|
|
|
Creation of the module object – that is, the implementation of
|
2015-05-07 21:58:19 -04:00
|
|
|
|
ExecutionLoader.create_module – is governed by the Py_mod_create slot.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
The Py_mod_create slot
|
|
|
|
|
......................
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
|
|
|
|
The Py_mod_create slot is used to support custom module subclasses.
|
|
|
|
|
The value pointer must point to a function with the following signature::
|
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
PyObject* (*PyModuleCreateFunction)(PyObject *spec, PyModuleDef *def)
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2022-01-21 06:03:51 -05:00
|
|
|
|
The function receives a ModuleSpec instance, as defined in :pep:`451`,
|
2015-05-07 21:58:19 -04:00
|
|
|
|
and the PyModuleDef structure.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
It should return a new module object, or set an error
|
|
|
|
|
and return NULL.
|
|
|
|
|
|
|
|
|
|
This function is not responsible for setting import-related attributes
|
2022-01-21 06:03:51 -05:00
|
|
|
|
specified in :pep:`451#attributes` (such as ``__name__`` or
|
2015-04-16 11:49:43 -04:00
|
|
|
|
``__loader__``) on the new module.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
|
|
|
|
There is no requirement for the returned object to be an instance of
|
|
|
|
|
types.ModuleType. Any type can be used, as long as it supports setting and
|
2015-04-16 11:49:43 -04:00
|
|
|
|
getting attributes, including at least the import-related attributes.
|
2015-05-07 21:58:19 -04:00
|
|
|
|
However, only ModuleType instances support module-specific functionality
|
2015-05-21 08:48:38 -04:00
|
|
|
|
such as per-module state and processing of execution slots.
|
|
|
|
|
If something other than a ModuleType subclass is returned, no execution slots
|
|
|
|
|
may be defined; if any are, a SystemError is raised.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
|
|
|
|
Note that when this function is called, the module's entry in sys.modules
|
|
|
|
|
is not populated yet. Attempting to import the same module again
|
|
|
|
|
(possibly transitively), may lead to an infinite loop.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
Extension authors are advised to keep Py_mod_create minimal, an in particular
|
2015-03-13 08:41:41 -04:00
|
|
|
|
to not call user code from it.
|
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
Multiple Py_mod_create slots may not be specified. If they are, import
|
2015-05-07 21:58:19 -04:00
|
|
|
|
will fail with SystemError.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-04-17 14:05:59 -04:00
|
|
|
|
If Py_mod_create is not specified, the import machinery will create a normal
|
2015-05-18 10:27:02 -04:00
|
|
|
|
module object using PyModule_New. The name is taken from *spec*.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
Post-creation steps
|
|
|
|
|
...................
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
If the Py_mod_create function returns an instance of types.ModuleType
|
2015-05-18 10:27:02 -04:00
|
|
|
|
or a subclass (or if a Py_mod_create slot is not present), the import
|
2015-05-20 07:27:55 -04:00
|
|
|
|
machinery will associate the PyModuleDef with the module.
|
|
|
|
|
This also makes the PyModuleDef accessible to execution phase, the
|
|
|
|
|
PyModule_GetDef function, and garbage collection routines (traverse,
|
|
|
|
|
clear, free).
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
If the Py_mod_create function does not return a module subclass, then m_size
|
2015-05-18 10:27:02 -04:00
|
|
|
|
must be 0, and m_traverse, m_clear and m_free must all be NULL.
|
2015-05-07 21:58:19 -04:00
|
|
|
|
Otherwise, SystemError is raised.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
Additionally, initial attributes specified in the PyModuleDef are set on the
|
|
|
|
|
module object, regardless of its type:
|
|
|
|
|
|
|
|
|
|
* The docstring is set from m_doc, if non-NULL.
|
|
|
|
|
* The module's functions are initialized from m_methods, if any.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
|
|
|
|
|
Module Execution Phase
|
|
|
|
|
----------------------
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
Module execution -- that is, the implementation of
|
|
|
|
|
ExecutionLoader.exec_module -- is governed by "execution slots".
|
|
|
|
|
This PEP only adds one, Py_mod_exec, but others may be added in the future.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-20 07:27:55 -04:00
|
|
|
|
The execution phase is done on the PyModuleDef associated with the module
|
|
|
|
|
object. For objects that are not a subclass of PyModule_Type (for which
|
2015-05-21 08:48:38 -04:00
|
|
|
|
PyModule_GetDef would fail), the execution phase is skipped.
|
2015-05-20 07:27:55 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
Execution slots may be specified multiple times, and are processed in the order
|
2015-04-16 11:49:43 -04:00
|
|
|
|
they appear in the slots array.
|
2015-05-07 21:58:19 -04:00
|
|
|
|
When using the default import machinery, they are processed after
|
2022-01-21 06:03:51 -05:00
|
|
|
|
import-related attributes specified in :pep:`451#attributes`
|
2015-04-16 11:49:43 -04:00
|
|
|
|
(such as ``__name__`` or ``__loader__``) are set and the module is added
|
|
|
|
|
to sys.modules.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
Pre-Execution steps
|
2015-05-21 08:48:38 -04:00
|
|
|
|
...................
|
2015-05-18 10:27:02 -04:00
|
|
|
|
|
|
|
|
|
Before processing the execution slots, per-module state is allocated for the
|
|
|
|
|
module. From this point on, per-module state is accessible through
|
|
|
|
|
PyModule_GetState.
|
|
|
|
|
|
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
The Py_mod_exec slot
|
|
|
|
|
....................
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-04-17 14:05:59 -04:00
|
|
|
|
The entry in this slot must point to a function with the following signature::
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
|
|
|
|
int (*PyModuleExecFunction)(PyObject* module)
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
It will be called to initialize a module. Usually, this amounts to
|
|
|
|
|
setting the module's initial attributes.
|
2015-05-07 21:58:19 -04:00
|
|
|
|
The "module" argument receives the module object to initialize.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
|
|
|
|
The function must return ``0`` on success, or, on error, set an exception and
|
|
|
|
|
return ``-1``.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-21 08:48:38 -04:00
|
|
|
|
If PyModuleExec replaces the module's entry in sys.modules, the new object
|
|
|
|
|
will be used and returned by importlib machinery after all execution slots
|
|
|
|
|
are processed. This is a feature of the import machinery itself.
|
|
|
|
|
The slots themselves are all processed using the module returned from the
|
|
|
|
|
creation phase; sys.modules is not consulted during the execution phase.
|
|
|
|
|
(Note that for extension modules, implementing Py_mod_create is usually
|
|
|
|
|
a better solution for using custom module objects.)
|
|
|
|
|
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
|
|
|
|
Legacy Init
|
|
|
|
|
-----------
|
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
The backwards-compatible single-phase initialization continues to be supported.
|
|
|
|
|
In this scheme, the PyInit function returns a fully initialized module rather
|
|
|
|
|
than a PyModuleDef object.
|
|
|
|
|
In this case, the PyInit hook implements the creation phase, and the execution
|
|
|
|
|
phase is a no-op.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-20 07:27:55 -04:00
|
|
|
|
Modules that need to work unchanged on older versions of Python should stick to
|
|
|
|
|
single-phase initialization, because the benefits it brings can't be
|
2015-05-18 10:27:02 -04:00
|
|
|
|
back-ported.
|
2015-05-20 07:27:55 -04:00
|
|
|
|
Here is an example of a module that supports multi-phase initialization,
|
|
|
|
|
and falls back to single-phase when compiled for an older version of CPython.
|
|
|
|
|
It is included mainly as an illustration of the changes needed to enable
|
|
|
|
|
multi-phase init::
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
#include <Python.h>
|
|
|
|
|
|
|
|
|
|
static int spam_exec(PyObject *module) {
|
|
|
|
|
PyModule_AddStringConstant(module, "food", "spam");
|
|
|
|
|
return 0;
|
|
|
|
|
}
|
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
#ifdef Py_mod_exec
|
2015-05-07 21:58:19 -04:00
|
|
|
|
static PyModuleDef_Slot spam_slots[] = {
|
|
|
|
|
{Py_mod_exec, spam_exec},
|
|
|
|
|
{0, NULL}
|
|
|
|
|
};
|
2015-05-18 10:27:02 -04:00
|
|
|
|
#endif
|
2015-05-07 21:58:19 -04:00
|
|
|
|
|
|
|
|
|
static PyModuleDef spam_def = {
|
|
|
|
|
PyModuleDef_HEAD_INIT, /* m_base */
|
|
|
|
|
"spam", /* m_name */
|
|
|
|
|
PyDoc_STR("Utilities for cooking spam"), /* m_doc */
|
|
|
|
|
0, /* m_size */
|
|
|
|
|
NULL, /* m_methods */
|
2015-05-18 10:27:02 -04:00
|
|
|
|
#ifdef Py_mod_exec
|
2015-05-07 21:58:19 -04:00
|
|
|
|
spam_slots, /* m_slots */
|
2015-05-18 10:27:02 -04:00
|
|
|
|
#else
|
|
|
|
|
NULL,
|
|
|
|
|
#endif
|
2015-05-07 21:58:19 -04:00
|
|
|
|
NULL, /* m_traverse */
|
|
|
|
|
NULL, /* m_clear */
|
|
|
|
|
NULL, /* m_free */
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
PyMODINIT_FUNC
|
|
|
|
|
PyInit_spam(void) {
|
2015-05-18 10:27:02 -04:00
|
|
|
|
#ifdef Py_mod_exec
|
|
|
|
|
return PyModuleDef_Init(&spam_def);
|
|
|
|
|
#else
|
2015-05-07 21:58:19 -04:00
|
|
|
|
PyObject *module;
|
|
|
|
|
module = PyModule_Create(&spam_def);
|
|
|
|
|
if (module == NULL) return NULL;
|
|
|
|
|
if (spam_exec(module) != 0) {
|
|
|
|
|
Py_DECREF(module);
|
|
|
|
|
return NULL;
|
|
|
|
|
}
|
|
|
|
|
return module;
|
2015-05-18 10:27:02 -04:00
|
|
|
|
#endif
|
2015-05-07 21:58:19 -04:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
Built-In modules
|
|
|
|
|
----------------
|
|
|
|
|
|
|
|
|
|
Any extension module can be used as a built-in module by linking it into
|
|
|
|
|
the executable, and including it in the inittab (either at runtime with
|
|
|
|
|
PyImport_AppendInittab, or at configuration time, using tools like *freeze*).
|
|
|
|
|
|
|
|
|
|
To keep this possibility, all changes to extension module loading introduced
|
|
|
|
|
in this PEP will also apply to built-in modules.
|
|
|
|
|
The only exception is non-ASCII module names, explained below.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
|
|
|
|
Subinterpreters and Interpreter Reloading
|
|
|
|
|
-----------------------------------------
|
|
|
|
|
|
|
|
|
|
Extensions using the new initialization scheme are expected to support
|
2015-05-20 07:27:55 -04:00
|
|
|
|
subinterpreters and multiple Py_Initialize/Py_Finalize cycles correctly,
|
|
|
|
|
avoiding the issues mentioned in Python documentation [#subinterpreter-docs]_.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
The mechanism is designed to make this easy, but care is still required
|
|
|
|
|
on the part of the extension author.
|
|
|
|
|
No user-defined functions, methods, or instances may leak to different
|
|
|
|
|
interpreters.
|
|
|
|
|
To achieve this, all module-level state should be kept in either the module
|
2015-04-16 11:49:43 -04:00
|
|
|
|
dict, or in the module object's storage reachable by PyModule_GetState.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
A simple rule of thumb is: Do not define any static data, except built-in types
|
|
|
|
|
with no mutable or user-settable class attributes.
|
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
|
|
|
|
|
Functions incompatible with multi-phase initialization
|
|
|
|
|
------------------------------------------------------
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
The PyModule_Create function will fail when used on a PyModuleDef structure
|
2015-05-18 10:27:02 -04:00
|
|
|
|
with a non-NULL *m_slots* pointer.
|
2015-05-07 21:58:19 -04:00
|
|
|
|
The function doesn't have access to the ModuleSpec object necessary for
|
2015-05-18 10:27:02 -04:00
|
|
|
|
multi-phase initialization.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
The PyState_FindModule function will return NULL, and PyState_AddModule
|
2015-05-18 10:27:02 -04:00
|
|
|
|
and PyState_RemoveModule will also fail on modules with non-NULL *m_slots*.
|
|
|
|
|
PyState registration is disabled because multiple module objects may be created
|
|
|
|
|
from the same PyModuleDef.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
Module state and C-level callbacks
|
|
|
|
|
----------------------------------
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
Due to the unavailability of PyState_FindModule, any function that needs access
|
|
|
|
|
to module-level state (including functions, classes or exceptions defined at
|
|
|
|
|
the module level) must receive a reference to the module object (or the
|
|
|
|
|
particular object it needs), either directly or indirectly.
|
|
|
|
|
This is currently difficult in two situations:
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
* Methods of classes, which receive a reference to the class, but not to
|
|
|
|
|
the class's module
|
|
|
|
|
* Libraries with C-level callbacks, unless the callbacks can receive custom
|
2015-05-18 10:27:02 -04:00
|
|
|
|
data set at callback registration
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
Fixing these cases is outside of the scope of this PEP, but will be needed for
|
|
|
|
|
the new mechanism to be useful to all modules. Proper fixes have been discussed
|
|
|
|
|
on the import-sig mailing list [#findmodule-discussion]_.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
As a rule of thumb, modules that rely on PyState_FindModule are, at the moment,
|
|
|
|
|
not good candidates for porting to the new mechanism.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
New Functions
|
|
|
|
|
-------------
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
A new function and macro implementing the module creation phase will be added.
|
2015-05-07 21:58:19 -04:00
|
|
|
|
These are similar to PyModule_Create and PyModule_Create2, except they
|
|
|
|
|
take an additional ModuleSpec argument, and handle module definitions with
|
|
|
|
|
non-NULL slots::
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
PyObject * PyModule_FromDefAndSpec(PyModuleDef *def, PyObject *spec)
|
|
|
|
|
PyObject * PyModule_FromDefAndSpec2(PyModuleDef *def, PyObject *spec,
|
|
|
|
|
int module_api_version)
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
A new function implementing the module execution phase will be added.
|
|
|
|
|
This allocates per-module state (if not allocated already), and *always*
|
|
|
|
|
processes execution slots. The import machinery calls this method when
|
|
|
|
|
a module is executed, unless the module is being reloaded::
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
PyAPI_FUNC(int) PyModule_ExecDef(PyObject *module, PyModuleDef *def)
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
Another function will be introduced to initialize a PyModuleDef object.
|
|
|
|
|
This idempotent function fills in the type, refcount, and module index.
|
|
|
|
|
It returns its argument cast to PyObject*, so it can be returned directly
|
|
|
|
|
from a PyInit function::
|
|
|
|
|
|
|
|
|
|
PyObject * PyModuleDef_Init(PyModuleDef *);
|
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
Additionally, two helpers will be added for setting the docstring and
|
|
|
|
|
methods on a module::
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
int PyModule_SetDocString(PyObject *, const char *)
|
|
|
|
|
int PyModule_AddFunctions(PyObject *, PyMethodDef *)
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Export Hook Name
|
|
|
|
|
----------------
|
|
|
|
|
|
|
|
|
|
As portable C identifiers are limited to ASCII, module names
|
2015-05-18 10:27:02 -04:00
|
|
|
|
must be encoded to form the PyInit hook name.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
|
|
|
|
For ASCII module names, the import hook is named
|
2015-05-18 10:27:02 -04:00
|
|
|
|
PyInit_<modulename>, where <modulename> is the name of the module.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
|
|
|
|
For module names containing non-ASCII characters, the import hook is named
|
2015-05-18 10:27:02 -04:00
|
|
|
|
PyInitU_<encodedname>, where the name is encoded using CPython's
|
2022-01-21 06:03:51 -05:00
|
|
|
|
"punycode" encoding (:rfc:`Punycode <3492>` with a lowercase suffix),
|
2015-04-16 11:49:43 -04:00
|
|
|
|
with hyphens ("-") replaced by underscores ("_").
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In Python::
|
|
|
|
|
|
|
|
|
|
def export_hook_name(name):
|
|
|
|
|
try:
|
2015-05-07 21:58:19 -04:00
|
|
|
|
suffix = b'_' + name.encode('ascii')
|
|
|
|
|
except UnicodeEncodeError:
|
|
|
|
|
suffix = b'U_' + name.encode('punycode').replace(b'-', b'_')
|
2015-05-18 10:27:02 -04:00
|
|
|
|
return b'PyInit' + suffix
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
|
|
|
|
Examples:
|
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
============= ===================
|
|
|
|
|
Module name Init hook name
|
|
|
|
|
============= ===================
|
|
|
|
|
spam PyInit_spam
|
|
|
|
|
lančmít PyInitU_lanmt_2sa6t
|
|
|
|
|
スパム PyInitU_zck5b2b
|
|
|
|
|
============= ===================
|
|
|
|
|
|
|
|
|
|
For modules with non-ASCII names, single-phase initialization is not supported.
|
|
|
|
|
|
|
|
|
|
In the initial implementation of this PEP, built-in modules with non-ASCII
|
|
|
|
|
names will not be supported.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
|
|
|
|
|
2015-03-13 08:41:41 -04:00
|
|
|
|
Module Reloading
|
|
|
|
|
----------------
|
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
Reloading an extension module using importlib.reload() will continue to
|
|
|
|
|
have no effect, except re-setting import-related attributes.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
Due to limitations in shared library loading (both dlopen on POSIX and
|
|
|
|
|
LoadModuleEx on Windows), it is not generally possible to load
|
2015-03-13 08:41:41 -04:00
|
|
|
|
a modified library after it has changed on disk.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
|
|
|
|
|
Use cases for reloading other than trying out a new version of the module
|
|
|
|
|
are too rare to require all module authors to keep reloading in mind.
|
|
|
|
|
If reload-like functionality is needed, authors can export a dedicated
|
|
|
|
|
function for it.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Multiple modules in one library
|
|
|
|
|
-------------------------------
|
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
To support multiple Python modules in one shared library, the library can
|
2015-05-18 10:27:02 -04:00
|
|
|
|
export additional PyInit* symbols besides the one that corresponds
|
2015-04-16 11:49:43 -04:00
|
|
|
|
to the library's filename.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
Note that this mechanism can currently only be used to *load* extra modules,
|
2015-05-20 07:27:55 -04:00
|
|
|
|
but not to *find* them. (This is a limitation of the loader mechanism,
|
|
|
|
|
which this PEP does not try to modify.)
|
|
|
|
|
To work around the lack of a suitable finder, code like the following
|
|
|
|
|
can be used::
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
import importlib.machinery
|
|
|
|
|
import importlib.util
|
|
|
|
|
loader = importlib.machinery.ExtensionFileLoader(name, path)
|
|
|
|
|
spec = importlib.util.spec_from_loader(name, loader)
|
2015-05-07 21:58:19 -04:00
|
|
|
|
module = importlib.util.module_from_spec(spec)
|
|
|
|
|
loader.exec_module(module)
|
|
|
|
|
return module
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
On platforms that support symbolic links, these may be used to install one
|
|
|
|
|
library under multiple names, exposing all exported modules to normal
|
|
|
|
|
import machinery.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Testing and initial implementations
|
|
|
|
|
-----------------------------------
|
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
For testing, a new built-in module ``_testmultiphase`` will be created.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
The library will export several additional modules using the mechanism
|
|
|
|
|
described in "Multiple modules in one library".
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
The ``_testcapi`` module will be unchanged, and will use single-phase
|
|
|
|
|
initialization indefinitely (or until it is no longer supported).
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
The ``array`` and ``xx*`` modules will be converted to use multi-phase
|
|
|
|
|
initialization as part of the initial implementation.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
Summary of API Changes and Additions
|
2015-05-21 08:48:38 -04:00
|
|
|
|
====================================
|
2015-05-07 21:58:19 -04:00
|
|
|
|
|
|
|
|
|
New functions:
|
|
|
|
|
|
|
|
|
|
* PyModule_FromDefAndSpec (macro)
|
|
|
|
|
* PyModule_FromDefAndSpec2
|
|
|
|
|
* PyModule_ExecDef
|
|
|
|
|
* PyModule_SetDocString
|
|
|
|
|
* PyModule_AddFunctions
|
2015-05-18 10:27:02 -04:00
|
|
|
|
* PyModuleDef_Init
|
2015-05-07 21:58:19 -04:00
|
|
|
|
|
|
|
|
|
New macros:
|
|
|
|
|
|
|
|
|
|
* Py_mod_create
|
|
|
|
|
* Py_mod_exec
|
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
New types:
|
|
|
|
|
|
|
|
|
|
* PyModuleDef_Type will be exposed
|
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
New structures:
|
|
|
|
|
|
|
|
|
|
* PyModuleDef_Slot
|
|
|
|
|
|
2015-05-22 17:45:38 -04:00
|
|
|
|
Other changes:
|
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
PyModuleDef.m_reload changes to PyModuleDef.m_slots.
|
|
|
|
|
|
2015-05-22 17:45:38 -04:00
|
|
|
|
``BuiltinImporter`` and ``ExtensionFileLoader`` will now implement
|
|
|
|
|
``create_module`` and ``exec_module``.
|
|
|
|
|
|
2015-05-20 07:27:55 -04:00
|
|
|
|
The internal ``_imp`` module will have backwards incompatible changes:
|
|
|
|
|
``create_builtin``, ``create_dynamic``, and ``exec_dynamic`` will be added;
|
|
|
|
|
``init_builtin``, ``load_dynamic`` will be removed.
|
|
|
|
|
|
|
|
|
|
The undocumented functions ``imp.load_dynamic`` and ``imp.init_builtin`` will
|
|
|
|
|
be replaced by backwards-compatible shims.
|
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
|
2015-05-21 08:48:38 -04:00
|
|
|
|
Backwards Compatibility
|
|
|
|
|
-----------------------
|
|
|
|
|
|
|
|
|
|
Existing modules will continue to be source- and binary-compatible with new
|
|
|
|
|
versions of Python.
|
|
|
|
|
Modules that use multi-phase initialization will not be compatible with
|
|
|
|
|
versions of Python that do not implement this PEP.
|
|
|
|
|
|
|
|
|
|
The functions ``init_builtin`` and ``load_dynamic`` will be removed from
|
|
|
|
|
the ``_imp`` module (but not from the ``imp`` module).
|
|
|
|
|
|
|
|
|
|
All changed loaders (``BuiltinImporter`` and ``ExtensionFileLoader``) will
|
|
|
|
|
remain backwards-compatible; the ``load_module`` method will be replaced by
|
|
|
|
|
a shim.
|
|
|
|
|
|
|
|
|
|
Internal functions of Python/import.c and Python/importdl.c will be removed.
|
|
|
|
|
(Specifically, these are ``_PyImport_GetDynLoadFunc``,
|
|
|
|
|
``_PyImport_GetDynLoadWindows``, and ``_PyImport_LoadDynamicModule``.)
|
|
|
|
|
|
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
Possible Future Extensions
|
|
|
|
|
==========================
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2022-01-21 06:03:51 -05:00
|
|
|
|
The slots mechanism, inspired by PyType_Slot from :pep:`384`,
|
2015-04-16 11:49:43 -04:00
|
|
|
|
allows later extensions.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
Some extension modules exports many constants; for example _ssl has
|
|
|
|
|
a long list of calls in the form::
|
|
|
|
|
|
|
|
|
|
PyModule_AddIntConstant(m, "SSL_ERROR_ZERO_RETURN",
|
|
|
|
|
PY_SSL_ERROR_ZERO_RETURN);
|
|
|
|
|
|
|
|
|
|
Converting this to a declarative list, similar to PyMethodDef,
|
|
|
|
|
would reduce boilerplate, and provide free error-checking which
|
|
|
|
|
is often missing.
|
|
|
|
|
|
|
|
|
|
String constants and types can be handled similarly.
|
|
|
|
|
(Note that non-default bases for types cannot be portably specified
|
|
|
|
|
statically; this case would need a Py_mod_exec function that runs
|
|
|
|
|
before the slots are added. The free error-checking would still be
|
|
|
|
|
beneficial, though.)
|
|
|
|
|
|
|
|
|
|
Another possibility is providing a "main" function that would be run
|
|
|
|
|
when the module is given to Python's -m switch.
|
|
|
|
|
For this to work, the runpy module will need to be modified to take
|
2022-01-21 06:03:51 -05:00
|
|
|
|
advantage of ModuleSpec-based loading introduced in :pep:`451`.
|
2015-04-16 11:49:43 -04:00
|
|
|
|
Also, it will be necessary to add a mechanism for setting up a module
|
|
|
|
|
according to slots it wasn't originally defined with.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Implementation
|
2015-03-13 08:41:41 -04:00
|
|
|
|
==============
|
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
Work-in-progress implementation is available in a Github repository [#gh-repo]_;
|
|
|
|
|
a patchset is at [#gh-patch]_.
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Previous Approaches
|
|
|
|
|
===================
|
|
|
|
|
|
|
|
|
|
Stefan Behnel's initial proto-PEP [#stefans_protopep]_
|
|
|
|
|
had a "PyInit_modulename" hook that would create a module class,
|
|
|
|
|
whose ``__init__`` would be then called to create the module.
|
2022-01-21 06:03:51 -05:00
|
|
|
|
This proposal did not correspond to the (then nonexistent) :pep:`451`,
|
2015-03-13 08:41:41 -04:00
|
|
|
|
where module creation and initialization is broken into distinct steps.
|
|
|
|
|
It also did not support loading an extension into pre-existing module objects.
|
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
Nick Coghlan proposed "Create" and "Exec" hooks, and wrote a prototype
|
2015-03-13 08:41:41 -04:00
|
|
|
|
implementation [#nicks-prototype]_.
|
2022-01-21 06:03:51 -05:00
|
|
|
|
At this time :pep:`451` was still not implemented, so the prototype
|
2015-03-13 08:41:41 -04:00
|
|
|
|
does not use ModuleSpec.
|
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
The original version of this PEP used Create and Exec hooks, and allowed
|
|
|
|
|
loading into arbitrary pre-constructed objects with Exec hook.
|
|
|
|
|
The proposal made extension module initialization closer to how Python modules
|
|
|
|
|
are initialized, but it was later recognized that this isn't an important goal.
|
|
|
|
|
The current PEP describes a simpler solution.
|
|
|
|
|
|
2015-05-18 10:27:02 -04:00
|
|
|
|
A further iteration used a "PyModuleExport" hook as an alternative to PyInit,
|
|
|
|
|
where PyInit was used for existing scheme, and PyModuleExport for multi-phase.
|
|
|
|
|
However, not being able to determine the hook name based on module name
|
|
|
|
|
complicated automatic generation of PyImport_Inittab by tools like freeze.
|
|
|
|
|
Keeping only the PyInit hook name, even if it's not entirely appropriate for
|
|
|
|
|
exporting a definition, yielded a much simpler solution.
|
|
|
|
|
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
|
|
|
|
References
|
|
|
|
|
==========
|
|
|
|
|
|
|
|
|
|
.. [#stefans_protopep]
|
|
|
|
|
https://mail.python.org/pipermail/python-dev/2013-August/128087.html
|
|
|
|
|
|
|
|
|
|
.. [#nicks-prototype]
|
|
|
|
|
https://mail.python.org/pipermail/python-dev/2013-August/128101.html
|
|
|
|
|
|
2015-04-16 11:49:43 -04:00
|
|
|
|
.. [#gh-repo]
|
|
|
|
|
https://github.com/encukou/cpython/commits/pep489
|
|
|
|
|
|
|
|
|
|
.. [#gh-patch]
|
|
|
|
|
https://github.com/encukou/cpython/compare/master...encukou:pep489.patch
|
|
|
|
|
|
2015-05-07 21:58:19 -04:00
|
|
|
|
.. [#findmodule-discussion]
|
|
|
|
|
https://mail.python.org/pipermail/import-sig/2015-April/000959.html
|
|
|
|
|
|
2015-05-20 07:27:55 -04:00
|
|
|
|
.. [#subinterpreter-docs]
|
|
|
|
|
https://docs.python.org/3/c-api/init.html#sub-interpreter-support
|
|
|
|
|
|
2015-03-13 08:41:41 -04:00
|
|
|
|
|
|
|
|
|
Copyright
|
|
|
|
|
=========
|
|
|
|
|
|
|
|
|
|
This document has been placed in the public domain.
|