PEP 573: SystemError->TypeError and misc copy-edits (GH-1264)
* report TypeError rather than SystemError when bad types are passed to new APIs * add PyType_FromModuleAndSpec to the list of new functions * wrap text at 80 columns * add C API docs links for thread-local storage and context variables * note in abstract that only the initial state lookup from slots is inherently slow, with subsequent lookups being amenable to caching * other minor wording tweaks
This commit is contained in:
Nick Coghlan
Petr Viktorin
parent
8dcf7fe49c
commit
34343da45d
140
pep-0573.rst
140
pep-0573.rst
|
|
@ -19,19 +19,24 @@ Post-History:
|
|||
Abstract
|
||||
========
|
||||
|
||||
This PEP proposes to add a way for CPython extension methods to access context such as
|
||||
the state of the modules they are defined in.
|
||||
This PEP proposes to add a way for CPython extension methods to access context,
|
||||
such as the state of the modules they are defined in.
|
||||
|
||||
This will allow extension methods to use direct pointer dereferences
|
||||
rather than PyState_FindModule for looking up module state, reducing or eliminating the
|
||||
performance cost of using module-scoped state over process global state.
|
||||
rather than PyState_FindModule for looking up module state, reducing or
|
||||
eliminating the performance cost of using module-scoped state over process
|
||||
global state.
|
||||
|
||||
This fixes one of the remaining roadblocks for adoption of PEP 3121 (Extension
|
||||
module initialization and finalization) and PEP 489
|
||||
(Multi-phase extension module initialization).
|
||||
|
||||
While this PEP takes an additional step towards fully solving the problems that PEP 3121 and PEP 489 started
|
||||
tackling, it does not attempt to resolve *all* remaining concerns. In particular, accessing the module state from slot methods (``nb_add``, etc) remains slower than accessing that state from other extension methods.
|
||||
While this PEP takes an additional step towards fully solving the problems that
|
||||
PEP 3121 and PEP 489 started tackling, it does not attempt to resolve *all*
|
||||
remaining concerns. In particular, at least the first access to the module state
|
||||
from slot methods (``nb_add``, etc) remains slower than accessing that state
|
||||
from other extension methods. Standard caching techniques can be used to speed
|
||||
up subsequent access.
|
||||
|
||||
|
||||
Terminology
|
||||
|
|
@ -58,12 +63,14 @@ Accessed by ``PyModule_GetState()``.
|
|||
Static Type
|
||||
-----------
|
||||
|
||||
A type object defined as a C-level static variable, i.e. a compiled-in type object.
|
||||
A type object defined as a C-level static variable, i.e. a compiled-in type
|
||||
object.
|
||||
|
||||
A static type needs to be shared between module instances and has no
|
||||
information of what module it belongs to.
|
||||
Static types do not have ``__dict__`` (although their instances might).
|
||||
|
||||
|
||||
Heap Type
|
||||
---------
|
||||
|
||||
|
|
@ -81,23 +88,23 @@ several advantages to extensions that implement it:
|
|||
module objects, which paves the way for extension module support for
|
||||
``runpy`` or for systems that enable extension module reloading.
|
||||
* Loading multiple modules from the same extension is possible, which
|
||||
makes testing module isolation (a key feature for proper sub-interpreter
|
||||
support) possible from a single interpreter.
|
||||
makes it possible to test module isolation (a key feature for proper
|
||||
sub-interpreter support) from a single interpreter.
|
||||
|
||||
The biggest hurdle for adoption of PEP 489 is allowing access to module state
|
||||
from methods of extension types.
|
||||
Currently, the way to access this state from extension methods is by looking up the module via
|
||||
``PyState_FindModule`` (in contrast to module level functions in extension modules, which
|
||||
receive a module reference as an argument).
|
||||
However, ``PyState_FindModule`` queries the thread-local state, making it relatively
|
||||
costly compared to C level process global access and consequently deterring module authors from using it.
|
||||
Currently, the way to access this state from extension methods is by looking up
|
||||
the module via ``PyState_FindModule`` (in contrast to module level functions in
|
||||
extension modules, which receive a module reference as an argument).
|
||||
However, ``PyState_FindModule`` queries the thread-local state, making it
|
||||
relatively costly compared to C level process global access and consequently
|
||||
deterring module authors from using it.
|
||||
|
||||
Also, ``PyState_FindModule`` relies on the assumption that in each
|
||||
subinterpreter, there is at most one module corresponding to
|
||||
a given ``PyModuleDef``. This does not align well with Python's import
|
||||
machinery. Since PEP 489 aimed to fix that, the assumption does
|
||||
not hold for modules that use multi-phase initialization, so
|
||||
``PyState_FindModule`` is unavailable for these modules.
|
||||
a given ``PyModuleDef``. This assumption does not hold for modules that use
|
||||
PEP 489's multi-phase initialization, so ``PyState_FindModule`` is unavailable
|
||||
for these modules.
|
||||
|
||||
A faster, safer way of accessing module-level state from extension methods
|
||||
is needed.
|
||||
|
|
@ -133,12 +140,17 @@ In Python code, the Python-level equivalents may be retrieved as::
|
|||
The defining class is not ``type(self)``, since ``type(self)`` might
|
||||
be a subclass of ``Foo``.
|
||||
|
||||
The statements marked (1) implicitly rely on name-based lookup via the function's ``__globals__``:
|
||||
either the ``Foo`` attribute to access the defining class and Python function object, or ``__name__`` to find the module object in ``sys.modules``.
|
||||
In Python code, this is feasible, as ``__globals__`` is set appropriately when the function definition is executed, and
|
||||
even if the namespace has been manipulated to return a different object, at worst an exception will be raised.
|
||||
The statements marked (1) implicitly rely on name-based lookup via the
|
||||
function's ``__globals__``: either the ``Foo`` attribute to access the defining
|
||||
class and Python function object, or ``__name__`` to find the module object in
|
||||
``sys.modules``.
|
||||
|
||||
The ``__class__`` closure, (2), is a safer way to get the defining class, but it still relies on ``__closure__`` being set appropriately.
|
||||
In Python code, this is feasible, as ``__globals__`` is set appropriately when
|
||||
the function definition is executed, and even if the namespace has been
|
||||
manipulated to return a different object, at worst an exception will be raised.
|
||||
|
||||
The ``__class__`` closure, (2), is a safer way to get the defining class, but it
|
||||
still relies on ``__closure__`` being set appropriately.
|
||||
|
||||
By contrast, extension methods are typically implemented as normal C functions.
|
||||
This means that they only have access to their arguments and C level thread-local
|
||||
|
|
@ -149,26 +161,34 @@ their shared state in C-level process globals, causing problems when:
|
|||
* reloading modules (e.g. to test conditional imports)
|
||||
* loading extension modules in subinterpreters
|
||||
|
||||
PEP 3121 attempted to resolve this by offering the ``PyState_FindModule`` API, but this still has significant problems when it comes to extension methods (rather than module level functions):
|
||||
PEP 3121 attempted to resolve this by offering the ``PyState_FindModule`` API,
|
||||
but this still has significant problems when it comes to extension methods
|
||||
(rather than module level functions):
|
||||
|
||||
* it is markedly slower than directly accessing C-level process-global state
|
||||
* there is still some inherent reliance on process global state that means it still doesn't reliably handle module reloading
|
||||
* there is still some inherent reliance on process global state that means it
|
||||
still doesn't reliably handle module reloading
|
||||
|
||||
It's also the case that when looking up a C-level struct such as module state,
|
||||
supplying an unexpected object layout can crash the interpreter, so it's
|
||||
significantly more important to ensure that extension methods receive the kind
|
||||
of object they expect.
|
||||
|
||||
It's also the case that when looking up a C-level struct such as module state, supplying
|
||||
an unexpected object layout can crash the interpreter, so it's significantly more important to ensure that extension
|
||||
methods receive the kind of object they expect.
|
||||
|
||||
Proposal
|
||||
========
|
||||
|
||||
Currently, a bound extension method (``PyCFunction`` or ``PyCFunctionWithKeywords``) receives only
|
||||
``self``, and (if applicable) the supplied positional and keyword arguments.
|
||||
Currently, a bound extension method (``PyCFunction`` or
|
||||
``PyCFunctionWithKeywords``) receives only ``self``, and (if applicable) the
|
||||
supplied positional and keyword arguments.
|
||||
|
||||
While module-level extension functions already receive access to the defining module object via their
|
||||
``self`` argument, methods of extension types don't have that luxury: they receive the bound instance
|
||||
via ``self``, and hence have no direct access to the defining class or the module level state.
|
||||
While module-level extension functions already receive access to the defining
|
||||
module object via their ``self`` argument, methods of extension types don't have
|
||||
that luxury: they receive the bound instance via ``self``, and hence have no
|
||||
direct access to the defining class or the module level state.
|
||||
|
||||
The additional module level context described above can be made available with two changes.
|
||||
The additional module level context described above can be made available with
|
||||
two changes.
|
||||
Both additions are optional; extension authors need to opt in to start
|
||||
using them:
|
||||
|
||||
|
|
@ -176,17 +196,16 @@ using them:
|
|||
|
||||
* Pass the defining class to the underlying C function.
|
||||
|
||||
The defining class is readily available at the time built-in
|
||||
The defining class is readily available at the time the built-in
|
||||
method object (``PyCFunctionObject``) is created, so it can be stored
|
||||
in a new struct that extends ``PyCFunctionObject``.
|
||||
|
||||
The module state can then be retrieved from the module object via
|
||||
``PyModule_GetState``.
|
||||
|
||||
Note that this proposal implies that any type whose method needs to access
|
||||
`per-module state`_ must be a heap type, rather than a static type.
|
||||
|
||||
This is necessary to support loading multiple module objects from a single
|
||||
Note that this proposal implies that any type whose methods need to access
|
||||
`per-module state`_ must be a heap type, rather than a static type. This is
|
||||
necessary to support loading multiple module objects from a single
|
||||
extension: a static type, as a C-level global, has no information about
|
||||
which module object it belongs to.
|
||||
|
||||
|
|
@ -201,17 +220,22 @@ simply add a new argument to pass in the defining class.
|
|||
Two possible solutions have been proposed to this problem:
|
||||
|
||||
* Look up the class through walking the MRO.
|
||||
This is potentially expensive, but will be useful if performance is not
|
||||
This is potentially expensive, but will be usable if performance is not
|
||||
a problem (such as when raising a module-level exception).
|
||||
* Storing a pointer to the defining class of each slot in a separate table,
|
||||
``__typeslots__`` [#typeslots-mail]_. This is technically feasible and fast,
|
||||
but quite invasive.
|
||||
|
||||
Due to the invasiveness of the latter approach, this PEP proposes adding an MRO walking
|
||||
helper for use in slot method implementations, deferring the more complex alternative
|
||||
as a potential future optimisation. Modules affected by this concern also have the
|
||||
option of using thread-local state or PEP 567 context variables, or else defining their
|
||||
own reload-friendly lookup caching scheme.
|
||||
Due to the invasiveness of the latter approach, this PEP proposes adding an MRO
|
||||
walking helper for use in slot method implementations, deferring the more complex
|
||||
alternative as a potential future optimisation.
|
||||
|
||||
Modules affected by this concern also have the option of using
|
||||
`thread-local state`_ or `PEP 567 context variables`_ as a caching mechanism, or
|
||||
else defining their own reload-friendly lookup caching scheme.
|
||||
|
||||
.. _thread-local state: https://docs.python.org/3/c-api/init.html#thread-local-storage-support
|
||||
.. _PEP 567 context variables: https://docs.python.org/3/c-api/contextvars.html
|
||||
|
||||
|
||||
Specification
|
||||
|
|
@ -237,12 +261,12 @@ This acts the same as ``PyType_FromSpecWithBases``, and additionally sets
|
|||
Additionally, an accessor, ``PyObject * PyType_GetModule(PyTypeObject *)``
|
||||
will be provided.
|
||||
It will return the ``ht_module`` if a heap type with module pointer set
|
||||
is passed in, otherwise it will set a SystemError and return NULL.
|
||||
is passed in, otherwise it will set ``TypeError`` and return NULL.
|
||||
|
||||
Usually, creating a class with ``ht_module`` set will create a reference
|
||||
cycle involving the class and the module.
|
||||
This is not a problem, as tearing down modules is not a performance-sensitive
|
||||
operation (and module-level functions typically also create reference cycles).
|
||||
operation, and module-level functions typically also create reference cycles.
|
||||
The existing "set all module globals to None" code that breaks function cycles
|
||||
through ``f_globals`` will also break the new cycles through ``ht_module``.
|
||||
|
||||
|
|
@ -325,9 +349,9 @@ will be implemented::
|
|||
The walker will go through bases of heap-allocated ``type``
|
||||
and search for class that defines ``func`` at its ``slot``.
|
||||
|
||||
The ``func`` needs not to be inherited by ``type``. The only requirement
|
||||
for the walker to find the defining class is that the defining class
|
||||
must be heap-allocated.
|
||||
The ``func`` does not need to be inherited by ``type`` (i.e. it may have been
|
||||
overridden in a subclass). The only requirement for the walker to find the
|
||||
defining class is that the defining class must be heap-allocated.
|
||||
|
||||
On failure, exception is set and NULL is returned.
|
||||
|
||||
|
|
@ -340,12 +364,13 @@ this easier, a helper will be added::
|
|||
|
||||
void *PyType_GetModuleState(PyObject *type)
|
||||
|
||||
This function takes a heap type and on success, it returns pointer to state of the
|
||||
module that the heap type belongs to.
|
||||
This function takes a heap type and on success, it returns pointer to the state
|
||||
of the module that the heap type belongs to.
|
||||
|
||||
On failure, two scenarios may occur. When a type without a module is passed in,
|
||||
``SystemError`` is set and ``NULL`` returned. If the module is found, pointer
|
||||
to the state, which may be ``NULL``, is returned without setting any exception.
|
||||
On failure, two scenarios may occur. When a non-type object, or a type without a
|
||||
module is passed in, ``TypeError`` is set and ``NULL`` returned. If the module
|
||||
is found, the pointer to the state, which may be ``NULL``, is returned without
|
||||
setting any exception.
|
||||
|
||||
|
||||
Modules Converted in the Initial Implementation
|
||||
|
|
@ -363,9 +388,10 @@ Summary of API Changes and Additions
|
|||
|
||||
New functions:
|
||||
|
||||
* ``PyType_FromModuleAndSpec``
|
||||
* ``PyType_GetModule``
|
||||
* ``PyType_DefiningTypeFromSlotFunc``
|
||||
* ``PyType_GetModuleState``
|
||||
* ``PyType_DefiningTypeFromSlotFunc``
|
||||
|
||||
New macros:
|
||||
|
||||
|
|
@ -388,7 +414,7 @@ Other changes:
|
|||
Backwards Compatibility
|
||||
=======================
|
||||
|
||||
Two new pointers are added to all heap types.
|
||||
One new pointer is added to all heap types.
|
||||
All other changes are adding new functions and structures,
|
||||
or changes to private implementation details.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue