iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

PEP 489: Updates from Petr Viktorin. 

Summary:

- PyModuleExport -> PyModuleDef (which brings us down to two slot types,
  create & exec)
- Removed "singleton modules"
- Stated that PyModule_Create, PyState_FindModule, PyState_AddModule,
  PyState_RemoveModule will not work on slots-based modules.
- Added a section on C-level callbacks
- Clarified that if PyModuleExport_* returns NULL, it's as if it wasn't
  defined (i.e. falls back to PyInit)
- Added API functions: PyModule_FromDefAndSpec, PyModule_ExecDef
- Added PyModule_AddMethods and PyModule_AddDocstring helpers
- Added PyMODEXPORT_FUNC macro for x-platform declarations of the export
  function
- Added summary of API changes
- Added example code for a backwards-compatible module
- Changed modules ported in the initial implementation to "array" and "xx*"
- Changed ImportErrors to SystemErrors in cases where the module is
  badly written (and to mirror what PyInit does now)
- Several typo fixes and clarifications
This commit is contained in:
Berker Peksag 2015-05-08 04:58:19 +03:00
parent 532c8f3c4c
commit 9891f4843b
1 changed files with 199 additions and 180 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

379
pep-0489.txt
View File

@ -116,34 +116,46 @@ is slightly different, see "Export Hook Name" below.)
If defined, this symbol must resolve to a C function with the following If defined, this symbol must resolve to a C function with the following
signature:: signature::
PyModuleExport* (*PyModuleExportFunction)(void) PyModuleDef* (*PyModuleExportFunction)(void)
The function must return a pointer to a PyModuleExport structure. For cross-platform compatibility, the function should be declared as::
PyMODEXPORT_FUNC PyModuleExport_<modulename>(void)
The function must return a pointer to a PyModuleDef structure.
This structure must be available for the lifetime of the module created from This structure must be available for the lifetime of the module created from
it – usually, it will be declared statically. it – usually, it will be declared statically.
The PyModuleExport structure describes the new module, similarly to Alternatively, this function can return NULL, in which case it is as if the
PEP 384's PyType_Spec for types. The structure is defined as:: symbol was not defined – see the "Legacy Init" section.
The PyModuleDef structure will be changed to contain a list of slots,
similarly to PEP 384's PyType_Spec for types.
To keep binary compatibility, and avoid needing to introduce a new structure
(which would introduce additional supporting functions and per-module storage),
the currently unused m_reload pointer of PyModuleDef will be changed to
hold the slots. The structures are defined as::
typedef struct { typedef struct {
int slot; int slot;
void *value; void *value;
} PyModuleExport_Slot; } PyModuleDef_Slot;
typedef struct { typedef struct PyModuleDef {
const char* doc; PyModuleDef_Base m_base;
int flags; const char* m_name;
PyModuleExport_Slot *slots; const char* m_doc;
} PyModuleExport; Py_ssize_t m_size;
PyMethodDef *m_methods;
PyModuleDef_Slot *m_slots; /* changed from `inquiry m_reload;` */
traverseproc m_traverse;
inquiry m_clear;
freefunc m_free;
} PyModuleDef;
The *doc* member specifies the module's docstring. The *m_slots* member must be either NULL, or point to an array of
PyModuleDef_Slot structures, terminated by a slot with id set to 0
The *flags* may currently be either 0 or ``PyModule_EXPORT_SINGLETON``, described (i.e. ``{0, NULL}``).
in "Singleton Modules" below.
Other flag values may be added in the future.
The *slots* points to an array of PyModuleExport_Slot structures, terminated
by a slot with id set to 0 (i.e. ``{0, NULL}``).
To specify a slot, a unique slot ID must be provided. To specify a slot, a unique slot ID must be provided.
New Python versions may introduce new slot IDs, but slot IDs will never be New Python versions may introduce new slot IDs, but slot IDs will never be
@ -153,54 +165,34 @@ throughout Python 3.x.
A slot's value pointer may not be NULL, unless specified otherwise in the A slot's value pointer may not be NULL, unless specified otherwise in the
slot's documentation. slot's documentation.
The following slots are available, and described later: The following slots are currently available, and described later:
* Py_mod_create * Py_mod_create
* Py_mod_statedef
* Py_mod_methods
* Py_mod_exec * Py_mod_exec
Unknown slot IDs will cause the import to fail with ImportError. Unknown slot IDs will cause the import to fail with SystemError.
.. note:: When using the new import mechanism, m_size must not be negative.
Also, the *m_name* field of PyModuleDef will not be unused during importing;
An alternate proposal is to use PyModuleDef instead of PyModuleExport, the module name will be taken from the ModuleSpec.
re-purposing the m_reload pointer to hold the slots::
typedef struct PyModuleDef {
PyModuleDef_Base m_base;
const char* m_name;
const char* m_doc;
Py_ssize_t m_size;
PyMethodDef *m_methods;
PyModuleExport_Slot* m_slots; /* changed from `inquiry m_reload;` */
traverseproc m_traverse;
inquiry m_clear;
freefunc m_free;
} PyModuleDef;
This would simplify both the implementation and the API, at the expense
of renaming a member of PyModuleDef, and re-purposing a function pointer as
a data pointer.
Creation Slots Module Creation
-------------- ---------------
The following slots affect module creation phase, i.e. they are hooks for Module creation – that is, the implementation of
ExecutionLoader.create_module. ExecutionLoader.create_module – is governed by the Py_mod_create slot.
They serve to describe creation of the module object itself.
Py_mod_create The Py_mod_create slot
............. ......................
The Py_mod_create slot is used to support custom module subclasses. The Py_mod_create slot is used to support custom module subclasses.
The value pointer must point to a function with the following signature:: The value pointer must point to a function with the following signature::
PyObject* (*PyModuleCreateFunction)(PyObject *spec, PyModuleExport *exp) PyObject* (*PyModuleCreateFunction)(PyObject *spec, PyModuleDef *def)
The function receives a ModuleSpec instance, as defined in PEP 451, The function receives a ModuleSpec instance, as defined in PEP 451,
and the PyModuleExport structure. and the PyModuleDef structure.
It should return a new module object, or set an error It should return a new module object, or set an error
and return NULL. and return NULL.
@ -211,15 +203,8 @@ specified in PEP 451 [#pep-0451-attributes]_ (such as ``__name__`` or
There is no requirement for the returned object to be an instance of There is no requirement for the returned object to be an instance of
types.ModuleType. Any type can be used, as long as it supports setting and types.ModuleType. Any type can be used, as long as it supports setting and
getting attributes, including at least the import-related attributes. getting attributes, including at least the import-related attributes.
However, only ModuleType instances support module-specific functionality
If a module instance is returned from Py_mod_create, the import machinery will such as per-module state.
store a pointer to PyModuleExport in the module object so that it may be
retrieved by PyModule_GetExport (described later).
.. note::
If PyModuleDef is used instead of PyModuleExport, the def is stored
instead, to be retrieved by PyModule_GetDef.
Note that when this function is called, the module's entry in sys.modules Note that when this function is called, the module's entry in sys.modules
is not populated yet. Attempting to import the same module again is not populated yet. Attempting to import the same module again
@ -228,70 +213,48 @@ Extension authors are advised to keep Py_mod_create minimal, an in particular
to not call user code from it. to not call user code from it.
Multiple Py_mod_create slots may not be specified. If they are, import Multiple Py_mod_create slots may not be specified. If they are, import
will fail with ImportError. will fail with SystemError.
If Py_mod_create is not specified, the import machinery will create a normal If Py_mod_create is not specified, the import machinery will create a normal
module object, as if by calling PyModule_Create. module object by PyModule_New. The name is taken from *spec*.
Py_mod_statedef Post-creation steps
............... ...................
The Py_mod_statedef slot is used to allocate per-module storage for C-level If the Py_mod_create function returns an instance of types.ModuleType
state. (or subclass), or if a Py_mod_create slot is not present, the import machinery
The value pointer must point to the following structure:: will do the following steps after the module is created:
typedef struct PyModule_StateDef { * If *m_size* is specified, per-module state is allocated and made accessible
int size; through PyModule_GetState
traverseproc traverse; * The PyModuleDef is associated with the module, making it accessible to
inquiry clear; PyModule_GetDef, and enabling the m_traverse, m_clear and m_free hooks.
freefunc free; * The docstring is set from m_doc.
} PyModule_StateDef; * The module's functions are initialized from m_methods.
The meaning of the members is the same as for the corresponding members in If the Py_mod_create function does not return a module subclass, then m_size
PyModuleDef. must be 0 or negative, and m_traverse, m_clear and m_free must all be NULL.
Otherwise, SystemError is raised.
Specifying multiple Py_mod_statedef slots, or specifying Py_mod_statedef
together with Py_mod_create, will cause the import to fail with ImportError.
.. note::
If PyModuleDef is reused, this information is taken from PyModuleDef,
so the slot is not necessary.
Execution slots Module Execution
--------------- ----------------
The following slots affect module "execution" phase, i.e. they are processed in Module execution -- that is, the implementation of
ExecutionLoader.exec_module. ExecutionLoader.exec_module -- is governed by "execution slots".
They serve to describe how the module is initialized – e.g. how it is populated This PEP only adds one, Py_mod_exec, but others may be added in the future.
with functions, types, or constants, and what import-time side effects
take place.
These slots may be specified multiple times, and are processed in the order Execution slots may be specified multiple times, and are processed in the order
they appear in the slots array. they appear in the slots array.
When using the default import machinery, they are processed after
When using the default import machinery, these slots are processed after
import-related attributes specified in PEP 451 [#pep-0451-attributes]_ import-related attributes specified in PEP 451 [#pep-0451-attributes]_
(such as ``__name__`` or ``__loader__``) are set and the module is added (such as ``__name__`` or ``__loader__``) are set and the module is added
to sys.modules. to sys.modules.
Py_mod_methods The Py_mod_exec slot
.............. ....................
This slot's value pointer must point to an array of PyMethodDef structures.
The specified methods are added to the module, like with PyModuleDef.m_methods.
.. note::
If PyModuleDef is reused this slot is unnecessary, since methods are
already included in PyModuleDef.
Py_mod_exec
...........
The entry in this slot must point to a function with the following signature:: The entry in this slot must point to a function with the following signature::
@ -299,12 +262,7 @@ The entry in this slot must point to a function with the following signature::
It will be called to initialize a module. Usually, this amounts to It will be called to initialize a module. Usually, this amounts to
setting the module's initial attributes. setting the module's initial attributes.
The "module" argument receives the module object to initialize.
The "module" argument receives the module object to initialize. This will
always be the module object created from the corresponding PyModuleExport.
When this function is called, import-related attributes (such as ``__spec__``)
will have been set, and the module has already been added to sys.modules.
If PyModuleExec replaces the module's entry in sys.modules, If PyModuleExec replaces the module's entry in sys.modules,
the new object will be used and returned by importlib machinery. the new object will be used and returned by importlib machinery.
@ -319,9 +277,9 @@ return ``-1``.
Legacy Init Legacy Init
----------- -----------
If the PyModuleExport function is not defined, the import machinery will try to If the PyModuleExport function is not defined, or if it returns NULL, the
initialize the module using the "PyInit_<modulename>" hook, import machinery will try to initialize the module using the
as described in PEP 3121. "PyInit_<modulename>" hook, as described in PEP 3121.
If the PyModuleExport function is defined, the PyInit function will be ignored. If the PyModuleExport function is defined, the PyInit function will be ignored.
Modules requiring compatibility with previous versions of CPython may implement Modules requiring compatibility with previous versions of CPython may implement
@ -330,17 +288,56 @@ the PyInit function in addition to the new hook.
Modules using the legacy init API will be initialized entirely in the Modules using the legacy init API will be initialized entirely in the
Loader.create_module step; Loader.exec_module will be a no-op. Loader.create_module step; Loader.exec_module will be a no-op.
.. XXX: Give example code for a backwards-compatible PyInit based on slots A module that supports older CPython versions can be coded as::
.. note:: #define Py_LIMITED_API
#include <Python.h>
If PyModuleDef is reused, implementing the PyInit function becomes easy: static int spam_exec(PyObject *module) {
PyModule_AddStringConstant(module, "food", "spam");
return 0;
}
* call PyModule_Create with the PyModuleDef (m_reload was ignored in static PyModuleDef_Slot spam_slots[] = {
previous Python versions, so the slots array will be ignored). {Py_mod_exec, spam_exec},
Alternatively, call the Py_mod_create function (keeping in mind that {0, NULL}
the spec is not available with PyInit). };
* call the Py_mod_exec function(s).
static PyModuleDef spam_def = {
PyModuleDef_HEAD_INIT, /* m_base */
"spam", /* m_name */
PyDoc_STR("Utilities for cooking spam"), /* m_doc */
0, /* m_size */
NULL, /* m_methods */
spam_slots, /* m_slots */
NULL, /* m_traverse */
NULL, /* m_clear */
NULL, /* m_free */
};
PyModuleDef* PyModuleExport_spam(void) {
return &spam_def;
}
PyMODINIT_FUNC
PyInit_spam(void) {
PyObject *module;
module = PyModule_Create(&spam_def);
if (module == NULL) return NULL;
if (spam_exec(module) != 0) {
Py_DECREF(module);
return NULL;
}
return module;
}
Note that this must be *compiled* on a new CPython version, but the resulting
shared library will be backwards compatible.
(Source-level compatibility is possible with preprocessor directives.)
If a Py_mod_create slot is used, PyInit should call its function instead of
PyModule_Create. Keep in mind that the ModuleSpec object is not available in
the legacy init scheme.
Subinterpreters and Interpreter Reloading Subinterpreters and Interpreter Reloading
@ -357,70 +354,63 @@ dict, or in the module object's storage reachable by PyModule_GetState.
A simple rule of thumb is: Do not define any static data, except built-in types A simple rule of thumb is: Do not define any static data, except built-in types
with no mutable or user-settable class attributes. with no mutable or user-settable class attributes.
Behavior of existing module creation functions
----------------------------------------------
PyModule_GetExport The PyModule_Create function will fail when used on a PyModuleDef structure
------------------ with a non-NULL m_slots pointer.
The function doesn't have access to the ModuleSpec object necessary for
"new style" module creation.
To retrieve the PyModuleExport structure used to create a module, The PyState_FindModule function will return NULL, and PyState_AddModule
a new function will be added:: and PyState_RemoveModule will fail with SystemError.
PyState registration is disabled because multiple module objects may be
PyModuleExport* PyModule_GetExport(PyObject *module) created from the same PyModuleDef.
The function returns NULL if the parameter is not a module object, or was not
created using PyModuleExport.
.. note::
This is unnecessary if PyModuleDef is reused: the existing
PyModule_GetDef can be used instead.
Singleton Modules Module state and C-level callbacks
----------------- ----------------------------------
Modules defined by PyModuleDef may be registered with PyState_AddModule, Due to the unavailability of PyState_FindModule, any function that needs access
and later retrieved with PyState_FindModule. to module-level state (including functions, classes or exceptions defined at
the module level) must receive a reference to the module object (or the
particular object it needs), either directly or indirectly.
This is currently difficult in two situations:
Under the new API, there is no one-to-one mapping between PyModuleSpec * Methods of classes, which receive a reference to the class, but not to
and the module created from it. the class's module
In particular, multiple modules may be loaded from the same description. * Libraries with C-level callbacks, unless the callbacks can receive custom
data set at cllback registration
This means that there is no "global" instance of a module object. Fixing these cases is outside of the scope of this PEP, but will be needed for
Any C-level callbacks that need access to the module state need to be passed the new mechanism to be useful to all modules. Proper fixes have been discussed
a reference to the module object, either directly or indirectly. on the import-sig mailing list [#findmodule-discussion]_.
As a rule of thumb, modules that rely on PyState_FindModule are, at the moment,
not good candidates for porting to the new mechanism.
However, there are some modules that really need to be only loaded once: New Functions
typically ones that wrap a C library with global state. -------------
These modules should set the PyModule_EXPORT_SINGLETON flag
in PyModuleExport.flags. When this flag is set, loading an additional
copy of the module after it has been loaded once will return the previously
loaded object.
This will be done on a low level, using _PyImport_FixupExtensionObject.
Additionally, the module will be automatically registered using
PyState_AddSingletonModule (see below) after execution slots are processed.
Singleton modules can be retrieved, registered or unregistered with A new function and macro will be added to implement module creation.
the interpreter state using three new functions, which parallel their These are similar to PyModule_Create and PyModule_Create2, except they
PyModuleDef counterparts, PyState_FindModule, PyState_AddModule, take an additional ModuleSpec argument, and handle module definitions with
and PyState_RemoveModule:: non-NULL slots::
PyObject* PyState_FindSingletonModule(PyModuleExport *exp) PyObject * PyModule_FromDefAndSpec(PyModuleDef *def, PyObject *spec)
int PyState_AddSingletonModule(PyObject *module, PyModuleExport *exp) PyObject * PyModule_FromDefAndSpec2(PyModuleDef *def, PyObject *spec,
int PyState_RemoveSingletonModule(PyModuleExport *exp) int module_api_version)
A new function will be added to run "execution slots" on a module::
.. note:: PyAPI_FUNC(int) PyModule_ExecDef(PyObject *module, PyModuleDef *def)
If PyModuleDef is used instead of PyModuleExport, the flag would be specified Additionally, two helpers will be added for setting the docstring and
as a slot with NULL value, i.e. ``{Py_mod_flag_singleton, NULL}``. methods on a module::
In this case, PyState_FindModule, PyState_AddModule and
PyState_RemoveModule can be used instead of the new functions.
.. note:: int PyModule_SetDocString(PyObject *, const char *)
int PyModule_AddFunctions(PyObject *, PyMethodDef *)
Another possibility is to use PyModuleDef_Base in PyModuleExport, and
have PyState_FindModule and friends work with either of the two structures.
Export Hook Name Export Hook Name
@ -442,10 +432,10 @@ In Python::
def export_hook_name(name): def export_hook_name(name):
try: try:
encoded = b'_' + name.encode('ascii') suffix = b'_' + name.encode('ascii')
except UnicodeDecodeError: except UnicodeEncodeError:
encoded = b'U_' + name.encode('punycode').replace(b'-', b'_') suffix = b'U_' + name.encode('punycode').replace(b'-', b'_')
return b'PyModuleExport' + encoded return b'PyModuleExport' + suffix
Examples: Examples:
@ -491,7 +481,9 @@ a module may be loaded with::
import importlib.util import importlib.util
loader = importlib.machinery.ExtensionFileLoader(name, path) loader = importlib.machinery.ExtensionFileLoader(name, path)
spec = importlib.util.spec_from_loader(name, loader) spec = importlib.util.spec_from_loader(name, loader)
return importlib.util.module_from_spec(spec) module = importlib.util.module_from_spec(spec)
loader.exec_module(module)
return module
On platforms that support symbolic links, these may be used to install one On platforms that support symbolic links, these may be used to install one
library under multiple names, exposing all exported modules to normal library under multiple names, exposing all exported modules to normal
@ -501,17 +493,41 @@ import machinery.
Testing and initial implementations Testing and initial implementations
----------------------------------- -----------------------------------
For testing, a new built-in module ``_testimportmodexport`` will be created. For testing, a new built-in module ``_testmoduleexport`` will be created.
The library will export several additional modules using the mechanism The library will export several additional modules using the mechanism
described in "Multiple modules in one library". described in "Multiple modules in one library".
The ``_testcapi`` module will be unchanged, and will use the old API The ``_testcapi`` module will be unchanged, and will use the old API
indefinitely (or until the old API is removed). indefinitely (or until the old API is removed).
The ``_csv`` and ``readline`` modules will be converted to the new API as The ``array`` and ``xx*`` modules will be converted to the new API as
part of the initial implementation. part of the initial implementation.
API Changes and Additions
-------------------------
New functions:
* PyModule_FromDefAndSpec (macro)
* PyModule_FromDefAndSpec2
* PyModule_ExecDef
* PyModule_SetDocString
* PyModule_AddFunctions
New macros:
* PyMODEXPORT_FUNC
* Py_mod_create
* Py_mod_exec
New structures:
* PyModuleDef_Slot
PyModuleDef.m_reload changes to PyModuleDef.m_slots.
Possible Future Extensions Possible Future Extensions
========================== ==========================
@ -595,6 +611,9 @@ References
.. [#gh-patch] .. [#gh-patch]
https://github.com/encukou/cpython/compare/master...encukou:pep489.patch https://github.com/encukou/cpython/compare/master...encukou:pep489.patch
.. [#findmodule-discussion]
https://mail.python.org/pipermail/import-sig/2015-April/000959.html
Copyright Copyright
========= =========