Update PEP-573 - Module state access
* Rewrite section on passing the "defining class" into C methods. * Remove proposal for immutable heap exception types. Let's leave those to another PEP * Restrict METH_METHOD to a single calling convention. This calling convention should be efficient, so also reduce the Optimization section. * Set names in monospaced font * Use `nargsf` for the vectorcall "number of args + flag" argument * Link "per-module state" to the terminology section * Adjust link to Marcel's current implementation * Other minor rewordings https://github.com/python/peps/pull/1159
This commit is contained in:
Stefan Behnel
Petr Viktorin
parent
53ac4d91a5
commit
b4495fd347
216
pep-0573.rst
216
pep-0573.rst
|
|
@ -30,10 +30,6 @@ This fixes one of the remaining roadblocks for adoption of PEP 3121 (Extension
|
|||
module initialization and finalization) and PEP 489
|
||||
(Multi-phase extension module initialization).
|
||||
|
||||
Additionally, support for easier creation of immutable exception classes is added.
|
||||
This removes the need for keeping per-module state if it would only be used
|
||||
for exception classes.
|
||||
|
||||
While this PEP takes an additional step towards fully solving the problems that PEP 3121 and PEP 489 started
|
||||
tackling, it does not attempt to resolve *all* remaining concerns. In particular, accessing the module state from slot methods (``nb_add``, etc) remains slower than accessing that state from other extension methods.
|
||||
|
||||
|
|
@ -47,6 +43,8 @@ Process-Global State
|
|||
C-level static variables. Since this is very low-level
|
||||
memory storage, it must be managed carefully.
|
||||
|
||||
.. _per-module state:
|
||||
|
||||
Per-module State
|
||||
----------------
|
||||
|
||||
|
|
@ -105,38 +103,6 @@ A faster, safer way of accessing module-level state from extension methods
|
|||
is needed.
|
||||
|
||||
|
||||
Immutable Exception Types
|
||||
-------------------------
|
||||
|
||||
For isolated modules to work, any class whose methods touch module state
|
||||
must be a heap type, so that each instance of a module can have its own
|
||||
type object. With the changes proposed in this PEP, heap type instances will
|
||||
have access to module state without global registration. But, to create
|
||||
instances of heap types, one will need the module state in order to
|
||||
get the type object corresponding to the appropriate module.
|
||||
In short, heap types are "viral" – anything that “touches” them must itself be
|
||||
a heap type.
|
||||
|
||||
Currently, most exception types, apart from the ones in ``builtins``, are
|
||||
heap types. This is likely simply because there is a convenient way
|
||||
to create them: ``PyErr_NewException``.
|
||||
Heap types generally have a mutable ``__dict__``.
|
||||
In most cases, this mutability is harmful. For example, exception types
|
||||
from the ``sqlite`` module are mutable and shared across subinterpreters.
|
||||
This allows "smuggling" values to other subinterpreters via attributes of
|
||||
``sqlite3.Error``.
|
||||
|
||||
Moreover, since raising exceptions is a common operation, and heap types
|
||||
will be "viral", ``PyErr_NewException`` will tend to "infect" the module
|
||||
with "heap type-ness" – at least if the module decides play well with
|
||||
subinterpreters/isolation.
|
||||
Many modules could go without module state
|
||||
entirely if the exception classes were immutable.
|
||||
|
||||
To solve this problem, a new function for creating immutable exception types
|
||||
is proposed.
|
||||
|
||||
|
||||
Background
|
||||
===========
|
||||
|
||||
|
|
@ -153,6 +119,7 @@ In Python code, the Python-level equivalents may be retrieved as::
|
|||
|
||||
import sys
|
||||
|
||||
class Foo:
|
||||
def meth(self):
|
||||
instance = self
|
||||
module_globals = globals()
|
||||
|
|
@ -217,11 +184,11 @@ The module state can then be retrieved from the module object via
|
|||
``PyModule_GetState``.
|
||||
|
||||
Note that this proposal implies that any type whose method needs to access
|
||||
per-module state must be a heap type, rather than a static type.
|
||||
`per-module state`_ must be a heap type, rather than a static type.
|
||||
|
||||
This is necessary to support loading multiple module objects from a single
|
||||
extension: a static type, as a C-level global, has no information about
|
||||
which module it belongs to.
|
||||
which module object it belongs to.
|
||||
|
||||
|
||||
Slot methods
|
||||
|
|
@ -247,22 +214,6 @@ option of using thread-local state or PEP 567 context variables, or else definin
|
|||
own reload-friendly lookup caching scheme.
|
||||
|
||||
|
||||
Immutable Exception Types
|
||||
-------------------------
|
||||
|
||||
To facilitate creating static exception classes, a new function is proposed:
|
||||
``PyErr_PrepareImmutableException``. It will work similarly to ``PyErr_NewExceptionWithDoc``
|
||||
but will take a ``PyTypeObject **`` pointer, which points to a ``PyTypeObject *`` that is
|
||||
either ``NULL`` or an initialized ``PyTypeObject``.
|
||||
This pointer may be declared in process-global state. The function will then
|
||||
allocate the object and will keep in mind that already existing exception
|
||||
should not be overwritten.
|
||||
|
||||
The extra indirection makes it possible to make ``PyErr_PrepareImmutableException``
|
||||
part of the stable ABI by having the Python interpreter, rather than extension code,
|
||||
allocate the ``PyTypeObject``.
|
||||
|
||||
|
||||
Specification
|
||||
=============
|
||||
|
||||
|
|
@ -299,15 +250,31 @@ through ``f_globals`` will also break the new cycles through ``ht_module``.
|
|||
Passing the defining class to extension methods
|
||||
-----------------------------------------------
|
||||
|
||||
A new style of C-level functions will be added to the current selection of
|
||||
``PyCFunction`` and ``PyCFunctionWithKeywords``::
|
||||
Since PEP 590 [#pep-590]_ was accepted for Python 3.8, ``PyCFunction``
|
||||
implements the vectorcall protocol.
|
||||
This PEP builds on top of PEP 590 to provide C implemented methods with
|
||||
context about their defining class (and thus their defining module).
|
||||
|
||||
A new signature flag, ``METH_METHOD``, will be added. Conceptually, it adds
|
||||
``defining_class`` to the function signature.
|
||||
To make the initial implementation easier, the flag can only be used as
|
||||
``(METH_FASTCALL | METH_KEYWORDS | METH_METHOD)``.
|
||||
(It can't be used with other flags like ``METH_O`` or bare ``METH_FASTCALL``,
|
||||
though it may be combined with ``METH_CLASS`` or ``METH_STATIC``).
|
||||
|
||||
A corresponding new C signature, ``PyCMethod``, is added to the ``PyCFunction``
|
||||
set of signatures::
|
||||
|
||||
PyObject *PyCMethod(PyObject *self,
|
||||
PyTypeObject *defining_class,
|
||||
PyObject *args, PyObject *kwargs)
|
||||
PyObject *const *args,
|
||||
size_t nargsf,
|
||||
PyObject *kwnames)
|
||||
|
||||
A new method object flag, ``METH_METHOD``, will be added to signal that
|
||||
the underlying C function is ``PyCMethod``.
|
||||
Additional combinations like ``(METH_VARARGS | METH_METHOD)`` may be added
|
||||
in the future (or even in the initial implementation of this PEP).
|
||||
However, ``METH_METHOD`` should always be an *additional* flag, i.e., the
|
||||
defining class should only be passed in if needed.
|
||||
|
||||
To hold the extra information, a new structure extending ``PyCFunctionObject``
|
||||
will be added::
|
||||
|
|
@ -317,15 +284,14 @@ will be added::
|
|||
PyTypeObject *mm_class; /* Passed as 'defining_class' arg to the C func */
|
||||
} PyCMethodObject;
|
||||
|
||||
To allow passing the defining class to the underlying C function, a change
|
||||
to private API is required, now ``_PyMethodDef_RawFastCallDict`` and
|
||||
``_PyMethodDef_RawFastCallKeywords`` will receive ``PyTypeObject *cls``
|
||||
as one of their arguments.
|
||||
The ``PyCFunction`` implementation will pass ``mm_class`` into a
|
||||
``PyCMethod`` C function when it finds the ``METH_METHOD`` flag being set.
|
||||
A new macro ``PyCFunction_GET_CLASS(cls)`` will be added for easier access
|
||||
to ``mm_class``.
|
||||
|
||||
A new macro ``PyCFunction_GET_CLASS(cls)`` will be added for easier access to mm_class.
|
||||
|
||||
Method construction and calling code and will be updated to honor
|
||||
``METH_METHOD``.
|
||||
C methods may continue to use the other ``METH_*`` signatures if they do
|
||||
not require access to their defining class/module.
|
||||
If ``METH_METHOD`` is not set, casting to ``PyCMethodObject`` is invalid.
|
||||
|
||||
|
||||
Argument Clinic
|
||||
|
|
@ -338,8 +304,8 @@ Each method may only have one argument using this converter, and it must
|
|||
appear after ``self``, or, if ``self`` is not used, as the first argument.
|
||||
The argument will be of type ``PyTypeObject *``.
|
||||
|
||||
When used, Argument Clinic will select ``METH_METHOD`` as the calling
|
||||
convention.
|
||||
When used, Argument Clinic will select
|
||||
``METH_FASTCALL | METH_KEYWORDS | METH_METHOD`` as the calling convention.
|
||||
The argument will not appear in ``__text_signature__``.
|
||||
|
||||
This will be compatible with ``__init__`` and ``__new__`` methods, where an
|
||||
|
|
@ -350,7 +316,7 @@ code to the user's function.
|
|||
Slot methods
|
||||
------------
|
||||
|
||||
To allow access to per-module state from slot methods, an MRO walker
|
||||
To allow access to `per-module state`_ from slot methods, an MRO walker
|
||||
will be implemented::
|
||||
|
||||
PyTypeObject *PyType_DefiningTypeFromSlotFunc(PyTypeObject *type,
|
||||
|
|
@ -359,54 +325,13 @@ will be implemented::
|
|||
The walker will go through bases of heap-allocated ``type``
|
||||
and search for class that defines ``func`` at its ``slot``.
|
||||
|
||||
The ``func`` needs not to be inherited by ``type``, only requirement
|
||||
The ``func`` needs not to be inherited by ``type``. The only requirement
|
||||
for the walker to find the defining class is that the defining class
|
||||
must be heap-allocated.
|
||||
|
||||
On failure, exception is set and NULL is returned.
|
||||
|
||||
|
||||
Static exceptions
|
||||
-----------------
|
||||
|
||||
A new function will be added::
|
||||
|
||||
int PyErr_PrepareImmutableException(PyTypeObject **exc,
|
||||
const char *name,
|
||||
const char *doc,
|
||||
PyObject *base)
|
||||
|
||||
Creates an immutable exception type which can be shared
|
||||
across multiple module objects.
|
||||
If the type already exists (determined by a process-global pointer,
|
||||
``*exc``), skip the initialization and only ``INCREF`` it.
|
||||
|
||||
If ``*exc`` is NULL, the function will
|
||||
allocate a new exception type and initialize it using given parameters
|
||||
the same way ``PyType_FromSpecAndBases`` would.
|
||||
The ``doc`` and ``base`` arguments may be ``NULL``, defaulting to a
|
||||
missing docstring and ``PyExc_Exception`` base class, respectively.
|
||||
The exception type's ``tp_flags`` will be set to values common to
|
||||
built-in exceptions and the ``Py_TPFLAGS_HEAP_IMMUTABLE`` flag (see below)
|
||||
will be set.
|
||||
On failure, ``PyErr_PrepareImmutableException`` will set an exception
|
||||
and return -1.
|
||||
|
||||
If called with an initialized exception type (``*exc``
|
||||
is non-NULL), the function will do nothing but incref ``*exc``.
|
||||
|
||||
A new flag, ``Py_TPFLAGS_HEAP_IMMUTABLE``, will be added to prevent
|
||||
mutation of the type object. This makes it possible to
|
||||
share the object safely between multiple interpreters.
|
||||
This flag is checked in ``type_setattro`` and blocks
|
||||
setting of attributes when set, similar to built-in types.
|
||||
|
||||
A new pointer, ``ht_moduleptr``, will be added to heap types to store ``exc``.
|
||||
|
||||
On deinitialization of the exception type, ``*exc`` will be set to ``NULL``.
|
||||
This makes it safe for ``PyErr_PrepareImmutableException`` to check if
|
||||
the exception was already initialized.
|
||||
|
||||
PyType_offsets
|
||||
--------------
|
||||
|
||||
|
|
@ -429,8 +354,8 @@ structure containing the mentioned data.
|
|||
Helpers
|
||||
-------
|
||||
|
||||
Getting to per-module state from a heap type is a very common task. To make this
|
||||
easier, a helper will be added::
|
||||
Getting to `per-module state`_ from a heap type is a very common task. To make
|
||||
this easier, a helper will be added::
|
||||
|
||||
void *PyType_GetModuleState(PyObject *type)
|
||||
|
||||
|
|
@ -457,55 +382,45 @@ Summary of API Changes and Additions
|
|||
|
||||
New functions:
|
||||
|
||||
* PyType_GetModule
|
||||
* PyType_DefiningTypeFromSlotFunc
|
||||
* PyType_GetModuleState
|
||||
* PyErr_PrepareImmutableException
|
||||
* ``PyType_GetModule``
|
||||
* ``PyType_DefiningTypeFromSlotFunc``
|
||||
* ``PyType_GetModuleState``
|
||||
|
||||
New macros:
|
||||
|
||||
* PyCFunction_GET_CLASS
|
||||
* ``PyCFunction_GET_CLASS``
|
||||
|
||||
New types:
|
||||
|
||||
* PyCMethodObject
|
||||
* ``PyCMethodObject``
|
||||
|
||||
New structures:
|
||||
|
||||
* PyType_offsets
|
||||
* ``PyType_offsets``
|
||||
|
||||
Modified functions:
|
||||
|
||||
* _PyMethodDef_RawFastCallDict now receives ``PyTypeObject *cls``.
|
||||
* _PyMethodDef_RawFastCallKeywords now receives ``PyTypeObject *cls``.
|
||||
* ``_PyMethodDef_RawFastCallDict`` now receives ``PyTypeObject *cls``.
|
||||
* ``_PyMethodDef_RawFastCallKeywords`` now receives ``PyTypeObject *cls``.
|
||||
|
||||
Modified structures:
|
||||
|
||||
* _heaptypeobject - added ht_module and ht_moduleptr
|
||||
* _heaptypeobject - added ``ht_module`` and ``ht_moduleptr``
|
||||
|
||||
Other changes:
|
||||
|
||||
* METH_METHOD call flag
|
||||
* defining_class converter in clinic
|
||||
* Py_TPFLAGS_HEAP_IMMUTABLE flag
|
||||
* Py_offsets type spec slot
|
||||
* ``METH_METHOD`` call flag
|
||||
* ``defining_class`` converter in clinic
|
||||
* ``Py_TPFLAGS_HEAP_IMMUTABLE`` flag
|
||||
* ``Py_offsets`` type spec slot
|
||||
|
||||
|
||||
Backwards Compatibility
|
||||
=======================
|
||||
|
||||
Two new pointers are added to all heap types.
|
||||
All other changes are adding new functions, structures and a type flag.
|
||||
|
||||
The new ``PyErr_PrepareImmutableException`` function changes encourages
|
||||
modules to switch from using heap type Exception classes to immutable ones,
|
||||
and a number of modules will be switched in the initial implementation.
|
||||
This change will prevent adding class attributes to such types.
|
||||
For example, the following will raise AttributeError::
|
||||
|
||||
sqlite.OperationalError.foo = None
|
||||
|
||||
Instances and subclasses of such exceptions will not be affected.
|
||||
All other changes are adding new functions, structures and a type flag,
|
||||
or changes to private implementation details.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
|
@ -525,18 +440,20 @@ creating heap types significantly easier than calling
|
|||
``PyType_FromModuleAndSpec``.
|
||||
This is left to a future PEP.
|
||||
|
||||
It may be good to add a good way to create static exception types from the
|
||||
limited API. Such exception types could be shared between subinterpreters,
|
||||
but instantiated without needing specific module state.
|
||||
This is also left to possible future discussions.
|
||||
|
||||
|
||||
Optimization
|
||||
------------
|
||||
|
||||
CPython optimizes calls to methods that have restricted signatures,
|
||||
such as not allowing keyword arguments.
|
||||
As proposed here, methods defined with the ``METH_METHOD`` flag only support
|
||||
one specific signature.
|
||||
|
||||
As proposed here, methods defined with the ``METH_METHOD`` flag do not support
|
||||
these optimizations.
|
||||
|
||||
Optimized calls still have the option of accessing per-module state
|
||||
the same way slot methods do.
|
||||
If it turns out that other signatures are needed for performance reasons,
|
||||
they may be added.
|
||||
|
||||
|
||||
References
|
||||
|
|
@ -546,10 +463,13 @@ References
|
|||
(https://mail.python.org/pipermail/import-sig/2015-July/001035.html)
|
||||
|
||||
.. [#gh-repo]
|
||||
https://github.com/Traceur759/cpython/commits/pep-c
|
||||
https://github.com/Dormouse759/cpython/tree/pep-c-rebase_newer
|
||||
|
||||
.. [#gh-patch]
|
||||
https://github.com/Traceur759/cpython/compare/master...Traceur759:pep-c.patch
|
||||
https://github.com/Dormouse759/cpython/compare/master...Dormouse759:pep-c-rebase_newer
|
||||
|
||||
.. [#pep-590]
|
||||
https://www.python.org/dev/peps/pep-0590/
|
||||
|
||||
|
||||
Copyright
|
||||
|
|
|
|||
Loading…
Reference in New Issue