963 lines
38 KiB
ReStructuredText
963 lines
38 KiB
ReStructuredText
PEP: 575
|
||
Title: Unifying function/method classes
|
||
Author: Jeroen Demeyer <J.Demeyer@UGent.be>
|
||
Status: Draft
|
||
Type: Standards Track
|
||
Content-Type: text/x-rst
|
||
Created: 27-Mar-2018
|
||
Python-Version: 3.8
|
||
Post-History: 31-Mar-2018, 12-Apr-2018
|
||
|
||
|
||
Abstract
|
||
========
|
||
|
||
Reorganize the class hierarchy for functions and methods
|
||
with the goal of reducing the difference between
|
||
built-in functions (implemented in C) and Python functions.
|
||
Mainly, make built-in functions behave more like Python functions
|
||
without sacrificing performance.
|
||
|
||
A new base class ``base_function`` is introduced and the various function
|
||
classes, as well as ``method`` (renamed to ``bound_method``), inherit from it.
|
||
|
||
We also allow subclassing of some of these function classes.
|
||
|
||
Motivation
|
||
==========
|
||
|
||
Currently, CPython has two different function classes:
|
||
the first is Python functions, which is what you get
|
||
when defining a function with ``def`` or ``lambda``.
|
||
The second is built-in functions such as ``len``, ``isinstance`` or ``numpy.dot``.
|
||
These are implemented in C.
|
||
|
||
These two classes are completely independent with different functionality.
|
||
In particular, it is currently not possible to implement a function efficiently in C
|
||
(only built-in functions can do that)
|
||
while still allowing introspection like ``inspect.signature`` or ``inspect.getsourcefile``
|
||
(only Python functions can do that).
|
||
This is a problem for projects like Cython [#cython]_ that want to do exactly that.
|
||
|
||
In Cython, this was worked around by inventing a new function class called ``cyfunction``.
|
||
Unfortunately, a new function class creates problems:
|
||
the ``inspect`` module does not recognize such functions as being functions [#bpo30071]_
|
||
and the performance is worse
|
||
(CPython has specific optimizations for calling built-in functions).
|
||
|
||
A second motivation is more generally making built-in functions and methods
|
||
behave more like Python functions and methods.
|
||
For example, Python unbound methods are just functions but
|
||
unbound methods of extension types (e.g. ``dict.get``) are a distinct class.
|
||
Bound methods of Python classes have a ``__func__`` attribute,
|
||
bound methods of extension types do not.
|
||
|
||
New classes
|
||
===========
|
||
|
||
This is the new class hierarchy for functions and methods::
|
||
|
||
object
|
||
|
|
||
|
|
||
base_function
|
||
/ | \
|
||
/ | \
|
||
/ | defined_function
|
||
/ | \
|
||
builtin_function (*) | \
|
||
| function
|
||
|
|
||
bound_method (*)
|
||
|
||
The two classes marked with (*) do *not* allow subclassing;
|
||
the others do.
|
||
|
||
There is no difference between functions and unbound methods,
|
||
while bound methods are instances of ``bound_method``.
|
||
|
||
base_function
|
||
-------------
|
||
|
||
The class ``base_function`` becomes a new base class for all function types.
|
||
It behaves like the existing ``builtin_function_or_method``
|
||
with some differences:
|
||
|
||
#. It acts as a descriptor implementing ``__get__`` to turn a function into a method
|
||
if ``m_self`` is ``NULL``.
|
||
If ``m_self`` is not ``NULL``,
|
||
then this is a no-op: the existing function is returned instead.
|
||
|
||
#. A new read-only slot ``__objclass__``, represented in the C structure as ``m_objclass``.
|
||
If this attribute exists, it must be a class.
|
||
If so, the function must be called with ``self`` being an instance of that class.
|
||
This is mainly meant to support unbound methods of extension types,
|
||
replacing ``method_descriptor``.
|
||
Also ``__qualname__`` will use ``__objclass__`` as namespace
|
||
(instead of ``m_self`` before).
|
||
|
||
#. Argument Clinic [#clinic]_ is not supported.
|
||
|
||
#. The field ``ml_doc`` and the attributes ``__doc__`` and ``__text_signature__``
|
||
are gone.
|
||
|
||
#. A new flag ``METH_ARG0_FUNCTION`` for ``ml_flags``.
|
||
If this flag is set, the C function stored in ``ml_meth`` will be called with first argument
|
||
equal to the function object instead of ``m_self``.
|
||
|
||
#. A new flag ``METH_BINDING`` for ``ml_flags`` which only applies to
|
||
functions of a module (not methods of a class).
|
||
If this flag is set, then ``m_self`` will be set to ``NULL`` instead
|
||
of the module.
|
||
This allows the function to behave more like a Python function
|
||
as it enables ``__get__``.
|
||
This flag also enables `Self slicing`_.
|
||
|
||
#. A new flag ``METH_PYTHON`` for ``ml_flags``.
|
||
This flag indicates that this function should be treated as Python function.
|
||
Ideally, use of this flag should be avoided because it goes
|
||
against the duck typing philosophy.
|
||
It is still needed in a few places though, for example `Profiling`_.
|
||
|
||
#. A new flag ``METH_CUSTOM`` for ``ml_flags`` which prevents automatic
|
||
generation of a ``builtin_function``, see `Automatic creation of built-in functions`_.
|
||
|
||
The goal of ``base_function`` is that it supports all different ways
|
||
of calling functions and methods in just one structure.
|
||
For example, the new flag ``METH_ARG0_FUNCTION``
|
||
will be used by the implementation of Python functions.
|
||
|
||
It is not possible to directly create instances of ``base_function``
|
||
(``tp_new`` is ``NULL``).
|
||
However, it is legal for C code to manually create instances.
|
||
|
||
These are the relevant C structures::
|
||
|
||
PyTypeObject PyBaseFunction_Type;
|
||
|
||
typedef struct {
|
||
PyObject_HEAD
|
||
PyCFunctionDef *m_ml; /* Description of the C function to call */
|
||
PyObject *m_self; /* __self__: anything, can be NULL; readonly */
|
||
PyObject *m_module; /* __module__: anything */
|
||
PyObject *m_weakreflist; /* List of weak references */
|
||
PyTypeObject *m_objclass; /* __objclass__: type or NULL; readonly */
|
||
} PyBaseFunctionObject;
|
||
|
||
typedef struct {
|
||
const char *ml_name; /* The name of the built-in function/method */
|
||
PyCFunction ml_meth; /* The C function that implements it */
|
||
int ml_flags; /* Combination of METH_xxx flags, which mostly
|
||
describe the args expected by the C func */
|
||
} PyCFunctionDef;
|
||
|
||
Subclasses may extend ``PyCFunctionDef`` with extra fields.
|
||
|
||
The Python attribute ``__self__`` returns ``m_self``,
|
||
except if ``METH_STATIC`` is set.
|
||
In that case, ``None`` is returned.
|
||
If ``m_self`` is ``NULL``, then there is no ``__self__`` attribute at all.
|
||
For that reason, we write either ``m_self`` or ``__self__`` in this PEP
|
||
with slightly different meanings.
|
||
|
||
builtin_function
|
||
----------------
|
||
|
||
This is a copy of ``base_function``, with the following differences:
|
||
|
||
#. ``m_ml`` points to a ``PyMethodDef`` structure,
|
||
extending ``PyCFunctionDef`` with an additional ``ml_doc``
|
||
field to implement ``__doc__`` and ``__text_signature__``
|
||
as read-only attributes::
|
||
|
||
typedef struct {
|
||
const char *ml_name;
|
||
PyCFunction ml_meth;
|
||
int ml_flags;
|
||
const char *ml_doc;
|
||
} PyMethodDef;
|
||
|
||
Note that ``PyMethodDef`` is part of the Python Stable ABI [#ABI]_
|
||
and it is used by most extension modules,
|
||
so we absolutely cannot change this structure.
|
||
|
||
#. Argument Clinic [#clinic]_ is supported.
|
||
|
||
The type object is ``PyTypeObject PyCFunction_Type``
|
||
and we define ``PyCFunctionObject`` as alias of ``PyBaseFunctionObject``.
|
||
|
||
defined_function
|
||
----------------
|
||
|
||
The class ``defined_function`` (a subclass of ``base_function``) adds
|
||
support for various standard attributes which are used in ``inspect``.
|
||
This would be a good class to use for auto-generated C code, for example produced by Cython [#cython]_.
|
||
|
||
The layout of the C structure is as follows::
|
||
|
||
PyTypeObject PyDefinedFunction_Type;
|
||
|
||
typedef struct {
|
||
PyBaseFunctionObject base;
|
||
PyObject *func_name; /* __name__: string */
|
||
PyObject *func_qualname; /* __qualname__: string */
|
||
PyObject *func_doc; /* __doc__: can be anything or NULL */
|
||
PyObject *func_code; /* __code__: code */
|
||
PyObject *func_globals; /* __globals__: anything; readonly */
|
||
PyObject *func_defaults; /* __defaults__: tuple or NULL */
|
||
PyObject *func_kwdefaults; /* __kwdefaults__: dict or NULL */
|
||
PyObject *func_closure; /* __closure__: tuple of cell objects or NULL; readonly */
|
||
PyObject *func_annotations; /* __annotations__: dict or NULL */
|
||
PyObject *func_dict; /* __dict__: dict or NULL */
|
||
} PyDefinedFunctionObject;
|
||
|
||
This class adds various slots like ``__doc__`` and ``__code__`` to access the C attributes.
|
||
The slot ``__name__`` returns ``func_name``.
|
||
When setting ``__name__``, also ``base.m_ml.ml_name`` is updated
|
||
with the UTF-8 encoded name.
|
||
|
||
None of the attributes is required to be meaningful.
|
||
In particular, ``__code__`` may not be a working code object,
|
||
possibly only a few fields may be filled in.
|
||
And ``__defaults__`` is not required to be used for calling the function.
|
||
|
||
Apart from adding these extra attributes,
|
||
``defined_function`` behaves exactly the same as ``base_function``.
|
||
|
||
function
|
||
--------
|
||
|
||
This is the class meant for functions implemented in Python,
|
||
formerly known as ``function``.
|
||
Unlike the other function types,
|
||
instances of ``function`` can be created from Python code.
|
||
This is not changed, so we do not describe the details in this PEP.
|
||
|
||
The layout of the C structure is almost the same as ``defined_function``::
|
||
|
||
PyTypeObject PyFunction_Type;
|
||
|
||
typedef struct {
|
||
PyBaseFunctionObject base;
|
||
PyObject *func_name; /* __name__: string */
|
||
PyObject *func_qualname; /* __qualname__: string */
|
||
PyObject *func_doc; /* __doc__: can be anything or NULL */
|
||
PyObject *func_code; /* __code__: code */
|
||
PyObject *func_defaults; /* __defaults__: tuple or NULL */
|
||
PyObject *func_kwdefaults; /* __kwdefaults__: dict or NULL */
|
||
PyObject *func_annotations; /* __annotations__: dict or NULL */
|
||
PyObject *func_globals; /* __globals__: anything; readonly */
|
||
PyObject *func_closure; /* __closure__: tuple of cell objects or NULL; readonly */
|
||
PyObject *func_dict; /* __dict__: dict or NULL */
|
||
PyCFunctionDef _ml; /* Storage for base.m_ml */
|
||
} PyFunctionObject;
|
||
|
||
The only difference is an ``_ml`` field
|
||
which reserves space to be used by ``base.m_ml``.
|
||
|
||
When constructing an instance of ``function`` from ``code`` and ``globals``,
|
||
an instance is created with ``base.m_ml = &_ml``,
|
||
``base.m_self = NULL`` and with the ``METH_PYTHON`` flag set.
|
||
|
||
To make subclassing easier, we also add a copy constructor:
|
||
if ``f`` is an instance of ``defined_function`` with the ``METH_PYTHON``
|
||
flag set, then ``types.FunctionType(f)`` copies ``f``.
|
||
|
||
bound_method
|
||
------------
|
||
|
||
The class ``bound_method`` is used for all bound methods,
|
||
regardless of the class of the underlying function.
|
||
It adds one new attribute on top of ``base_function``:
|
||
``__func__`` points to that function.
|
||
|
||
``bound_method`` replaces the old ``method`` class
|
||
which was used only for Python functions bound as method.
|
||
|
||
There is a complication because we want to allow
|
||
constructing a method from an arbitrary callable.
|
||
This may be an already-bound method or simply not an instance of ``base_function``.
|
||
Therefore, in practice there are two kinds of methods:
|
||
|
||
- For arbitrary callables, we use a single fixed ``PyCFunctionDef``
|
||
structure with the ``METH_ARG0_FUNCTION`` flag set.
|
||
The C function then calls ``__func__`` with the correct arguments.
|
||
|
||
- For methods which bind instances of ``base_function``
|
||
(more precisely, which have the ``Py_TPFLAGS_BASEFUNCTION`` flag set)
|
||
that allow self slicing,
|
||
we instead use the ``PyCFunctionDef`` from the original function.
|
||
In this case, the ``__func__`` attribute is only used to implement
|
||
various attributes but not for calling the method.
|
||
|
||
When constructing a new method from a ``base_function``,
|
||
we check that the ``self`` object is an instance of ``__objclass__``
|
||
(if such a class was specified) and raise a ``TypeError`` otherwise.
|
||
|
||
The C structure is::
|
||
|
||
PyTypeObject PyMethod_Type;
|
||
|
||
typedef struct {
|
||
PyBaseFunctionObject base;
|
||
PyObject *im_func; /* __func__: function implementing the method; readonly */
|
||
} PyMethodObject;
|
||
|
||
|
||
|
||
Calling base_function instances
|
||
===============================
|
||
|
||
We specify the implementation of ``__call__`` for instances of ``base_function``.
|
||
|
||
__objclass__
|
||
------------
|
||
|
||
First of all, if the function has an ``__objclass__`` attribute but
|
||
``m_self`` is ``NULL`` (this is the case for unbound methods of extension types),
|
||
then the function must be called with at least one positional argument
|
||
and the first (typically called ``self``) must be an instance of ``__objclass__``.
|
||
If not, a ``TypeError`` is raised.
|
||
|
||
Flags
|
||
-----
|
||
|
||
For convenience, we define a new constant:
|
||
``METH_CALLSIGNATURE`` combines all flags from ``PyCFunctionDef.ml_flags``
|
||
which specify the signature of the C function to be called.
|
||
It is equal to ::
|
||
|
||
METH_VARARGS | METH_FASTCALL | METH_NOARGS | METH_O | METH_KEYWORDS
|
||
|
||
Exactly one of the first four flags above must be set
|
||
and only ``METH_VARARGS`` and ``METH_FASTCALL`` may be combined with ``METH_KEYWORDS``.
|
||
Violating these rules is undefined behaviour.
|
||
|
||
There is one new flag which affects calling functions,
|
||
namely ``METH_ARG0_FUNCTION``.
|
||
Some flags are already documented in [#methoddoc]_.
|
||
We explain the other two shortly.
|
||
|
||
Self slicing
|
||
------------
|
||
|
||
If the function has ``m_self == NULL``
|
||
and the flag ``METH_ARG0_FUNCTION`` is not set,
|
||
then the first positional argument (if any)
|
||
is removed from ``*args`` and instead passed as first argument to the C function.
|
||
Effectively, the first positional argument is treated as ``__self__``.
|
||
This process is called "self slicing" and is meant to support unbound methods.
|
||
This does not affect keyword arguments.
|
||
|
||
METH_FASTCALL
|
||
-------------
|
||
|
||
This is an existing but undocumented flag.
|
||
We suggest to officially support and document it.
|
||
|
||
If the flag ``METH_FASTCALL`` is set without ``METH_KEYWORDS``,
|
||
then the ``ml_meth`` field is of type ``PyCFunctionFast``
|
||
which takes the arguments ``(PyObject *arg0, PyObject *const *args, Py_ssize_t nargs)``.
|
||
Such a function takes only positional arguments and they are passed as plain C array
|
||
``args`` of length ``nargs``.
|
||
|
||
If the flags ``METH_FASTCALL | METH_KEYWORDS`` are set,
|
||
then the ``ml_meth`` field is of type ``PyCFunctionFastWithKeywords``
|
||
which takes the arguments ``(PyObject *arg0, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)``.
|
||
The positional arguments are passed as C array ``args`` of length ``nargs``.
|
||
The *values* of the keyword arguments follow in that array,
|
||
starting at position ``nargs``.
|
||
The *keys* (names) of the keyword arguments are passed as a ``tuple`` in ``kwnames``.
|
||
As an example, assume that 3 positional and 2 keyword arguments are given.
|
||
Then ``args`` is an array of length 3 + 2 = 5, ``nargs`` equals 3 and ``kwnames`` is a 2-tuple.
|
||
|
||
METH_ARG0_FUNCTION
|
||
------------------
|
||
|
||
If this flag is set, then the first argument to the C function
|
||
is the function itself (the ``base_function`` instance) instead of ``m_self``.
|
||
In this case, the C function should deal with ``__self__``
|
||
by getting it from the function, for example using ``PyBaseFunction_GET_SELF``.
|
||
|
||
|
||
Automatic creation of built-in functions
|
||
========================================
|
||
|
||
Python automatically generates instances of ``builtin_function``
|
||
for extension types (using the ``PyTypeObject.tp_methods`` field) and modules
|
||
(using the ``PyModuleDef.m_methods`` field).
|
||
The arrays ``PyTypeObject.tp_methods`` and ``PyModuleDef.m_methods``
|
||
must be arrays of ``PyMethodDef`` structures.
|
||
|
||
If the ``METH_CUSTOM`` flag is set for an element of such an array,
|
||
then no ``builtin_function`` will be generated.
|
||
This allows an application to customize the creation of functions
|
||
in an extension type or module.
|
||
If ``METH_CUSTOM`` is set, then ``METH_STATIC`` and ``METH_CLASS`` are ignored.
|
||
|
||
Built-in unbound methods
|
||
------------------------
|
||
|
||
The type of unbound methods changes from ``method_descriptor``
|
||
to ``builtin_function``.
|
||
The object which appears as unbound method is the same object which
|
||
appears in the class ``__dict__``.
|
||
Python automatically sets the ``__objclass__`` attribute.
|
||
|
||
Built-in functions of a module
|
||
------------------------------
|
||
|
||
For the case of functions of a module,
|
||
``__self__`` will be set to the module unless the flag ``METH_BINDING`` is set.
|
||
|
||
An important consequence is that such functions by default
|
||
do not become methods when used as attribute
|
||
(``base_function.__get__`` only does that if ``m_self`` was ``NULL``).
|
||
One could consider this a bug, but this was done for backwards compatibility reasons:
|
||
in an initial post on python-ideas [#proposal]_ the concensus was to keep this
|
||
misfeature of built-in functions.
|
||
|
||
However, to allow this anyway for specific or newly implemented
|
||
built-in functions, the ``METH_BINDING`` flag prevents setting ``__self__``.
|
||
|
||
|
||
Further changes
|
||
===============
|
||
|
||
New type flag
|
||
-------------
|
||
|
||
A new ``PyTypeObject`` flag (for ``tp_flags``) is added:
|
||
``Py_TPFLAGS_BASEFUNCTION`` to indicate that instances of this type are
|
||
functions which can be called as a ``base_function``.
|
||
In other words, subclasses of ``base_function``
|
||
which follow the implementation from `Calling base_function instances`_.
|
||
|
||
This is different from flags like ``Py_TPFLAGS_LIST_SUBCLASS``
|
||
because it indicates more than just a subclass:
|
||
it also indicates a default implementation of ``__call__``.
|
||
This flag is never inherited.
|
||
However, extension types can explicitly specify it if they
|
||
do not override ``__call__`` or if they override ``__call__`` in a compatible way.
|
||
The flag ``Py_TPFLAGS_BASEFUNCTION`` must never be set for a heap type
|
||
because that would not be safe (heap types can be changed dynamically).
|
||
|
||
C API functions
|
||
---------------
|
||
|
||
We list some relevant Python/C API macros and functions.
|
||
Some of these are existing (possibly changed) functions, some are new:
|
||
|
||
- ``int PyBaseFunction_CheckFast(PyObject *op)``: return true if ``op``
|
||
is an instance of a class with the ``Py_TPFLAGS_BASEFUNCTION`` set.
|
||
This is the function that you need to use to determine
|
||
whether you can safely access the ``base_function`` internals.
|
||
|
||
- ``int PyBaseFunction_Check(PyObject *op)``: return true if ``op``
|
||
is an instance of ``base_function``.
|
||
|
||
- ``PyObject *PyBaseFunction_New(PyTypeObject *cls, PyCFunctionDef *ml, PyObject *self, PyObject *module, PyTypeObject *objclass)``:
|
||
create a new instance of ``cls`` (which must be a subclass of ``base_function``)
|
||
from the given data.
|
||
|
||
- ``int PyCFunction_Check(PyObject *op)``: return true if ``op``
|
||
is an instance of ``builtin_function``.
|
||
|
||
- ``int PyCFunction_NewEx(PyMethodDef* ml, PyObject *self, PyObject* module)``:
|
||
create a new instance of ``builtin_function``.
|
||
As special case, if ``self`` is ``NULL``,
|
||
then set ``self = Py_None`` instead (for backwards compatibility).
|
||
|
||
- For many existing ``PyCFunction_...`` and ``PyMethod_`` functions,
|
||
we define a new function ``PyBaseFunction_...``
|
||
acting on ``base_function`` instances.
|
||
For backwards compatibility,
|
||
the old functions are kept as aliases of the new functions.
|
||
|
||
- ``int PyFunction_Check(PyObject *op)``: return true if ``op``
|
||
is an instance of ``defined_function``.
|
||
|
||
- ``PyObject *PyFunction_NewPython(PyTypeObject *cls, PyObject *code, PyObject *globals, PyObject *name, PyObject *qualname)``:
|
||
create a new instance of ``cls`` (which must be a sublass of ``function``)
|
||
from the given data.
|
||
|
||
- ``PyObject *PyFunction_New(PyObject *code, PyObject *globals)``:
|
||
create a new instance of ``function``.
|
||
|
||
- ``PyObject *PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname)``:
|
||
create a new instance of ``function``.
|
||
|
||
- ``PyObject *PyFunction_Copy(PyTypeObject *cls, PyObject *func)``:
|
||
create a new instance of ``cls`` (which must be a sublass of ``function``)
|
||
by copying a given ``defined_function``.
|
||
|
||
- All other existing ``PyFunction_...`` functions now act on ``defined_function``
|
||
instances (instead of ``function``).
|
||
|
||
Changes to the types module
|
||
---------------------------
|
||
|
||
Two types are added: ``types.BaseFunctionType`` corresponding to
|
||
``base_function`` and ``types.DefinedFunctionType`` corresponding to
|
||
``defined_function``.
|
||
|
||
Apart from that, no changes to the ``types`` module are made.
|
||
In particular, ``types.FunctionType`` refers to ``function``.
|
||
However, the actual types will change:
|
||
for example, ``types.BuiltinFunctionType`` will no longer be the same
|
||
as ``types.BuiltinMethodType``.
|
||
|
||
Changes to the inspect module
|
||
-----------------------------
|
||
|
||
``inspect.isbasefunction`` checks for an instance of ``base_function``.
|
||
|
||
``inspect.isfunction`` checks for an instance of ``defined_function``.
|
||
|
||
``inspect.isbuiltin`` checks for an instance of ``builtin_function``.
|
||
|
||
``inspect.isroutine`` checks ``isbasefunction`` or ``ismethoddescriptor``.
|
||
|
||
Note that bpo-33261 [#bpo33261]_ should be fixed first.
|
||
|
||
Profiling
|
||
---------
|
||
|
||
Currently, ``sys.setprofile`` supports ``c_call``, ``c_return`` and ``c_exception``
|
||
events for built-in functions.
|
||
These events are generated when calling or returning from a built-in function.
|
||
By contrast, the ``call`` and ``return`` events are generated by the function itself.
|
||
So nothing needs to change for the ``call`` and ``return`` events.
|
||
|
||
Since we no longer make a difference between C functions and Python functions,
|
||
we need to prevent the ``c_*`` events for Python functions.
|
||
This is done by not generating those events if the
|
||
``METH_PYTHON`` flag in ``ml_flags`` is set.
|
||
|
||
User flags in PyCFunctionDef.ml_flags
|
||
----------------------------------------
|
||
|
||
8 consecutive bits in ``ml_flags`` are reserved for the "user",
|
||
meaning the person or program who implemented the function.
|
||
These are ``METH_USR0``, ..., ``METH_USR7``.
|
||
Python will ignore these flags.
|
||
|
||
It should be clear that different users may use these flags
|
||
for different purposes, so users should only look at those flags in
|
||
functions that they implemented (for example, by looking for those flags
|
||
in the ``tp_methods`` array of an extension type).
|
||
|
||
|
||
Non-CPython implementations
|
||
===========================
|
||
|
||
For other implementations of Python apart from CPython,
|
||
only the classes ``base_function``, ``bound_method`` and ``function`` are required.
|
||
The latter two are the only classes which can be instantiated directly
|
||
from the Python interpreter.
|
||
We require ``base_function`` for consistency but we put no requirements on it:
|
||
it is acceptable if this is just a copy of ``object``.
|
||
Support for the new ``__objclass__`` attribute is not required.
|
||
If there is no ``defined_function`` type,
|
||
then ``types.DefinedFunctionType`` should be an alias of ``types.FunctionType``.
|
||
|
||
|
||
Rationale
|
||
=========
|
||
|
||
Why not simply change existing classes?
|
||
---------------------------------------
|
||
|
||
One could try to solve the problem not by introducing a new ``base_function``
|
||
class and changing the class hierarchy, but by just changing existing classes.
|
||
|
||
That might look like a simpler solution but it is not:
|
||
it would require introspection support for 3 distinct classes:
|
||
``function``, ``builtin_function_or_method`` and ``method_descriptor``.
|
||
In the current PEP, there is only a single class where introspection needs
|
||
to be implemented.
|
||
It is also not clear how this would interact with ``__text_signature__``.
|
||
Having two independent kinds of ``inspect.signature`` support on the same
|
||
class sounds like asking for problems.
|
||
|
||
And this would not fix some of the other differences between built-in functions
|
||
and Python functions that were mentioned in the `Motivation`_.
|
||
|
||
Why __text_signature__ is not a solution
|
||
----------------------------------------
|
||
|
||
Built-in functions have an attribute ``__text_signature__``,
|
||
which gives the signature of the function as plain text.
|
||
The default values are evaluated by ``ast.literal_eval``.
|
||
Because of this, it supports only a small number of standard Python classes
|
||
and not arbitrary Python objects.
|
||
|
||
And even if ``__text_signature__`` would allow arbitrary signatures somehow,
|
||
that is only one piece of introspection:
|
||
it does not help with ``inspect.getsourcefile`` for example.
|
||
|
||
defined_function versus function
|
||
--------------------------------
|
||
|
||
In many places, a decision needs to be made whether the old ``function`` class
|
||
should be replaced by ``defined_function`` or the new ``function`` class.
|
||
This is done by thinking of the most likely use case:
|
||
|
||
1. ``types.FunctionType`` refers to ``function`` because that
|
||
type might be used to construct instances using ``types.FunctionType(...)``.
|
||
|
||
2. ``inspect.isfunction()`` refers to ``defined_function``
|
||
because this is the class where introspection is supported.
|
||
|
||
3. The C API functions ``PyFunction_New...``
|
||
refer to ``function`` simply because one cannot create instances
|
||
of ``defined_function``.
|
||
|
||
4. The C API functions ``PyFunction_Check`` and ``PyFunction_Get/Set...``
|
||
refer to ``defined_function`` because all attributes exist for instances of ``defined_function``.
|
||
|
||
Scope of this PEP: which classes are involved?
|
||
----------------------------------------------
|
||
|
||
The main motivation of this PEP is fixing function classes,
|
||
so we certainly want to unify the existing classes
|
||
``builtin_function_or_method`` and ``function``.
|
||
|
||
Since built-in functions and methods have the same class,
|
||
it seems natural to include bound methods too.
|
||
And since there are no "unbound methods" for Python functions,
|
||
it makes sense to get rid of unbound methods for extension types.
|
||
|
||
For now, no changes are made to the classes ``staticmethod``,
|
||
``classmethod`` and ``classmethod_descriptor``.
|
||
It would certainly make sense to put these in the ``base_function``
|
||
class hierarchy and unify ``classmethod`` and ``classmethod_descriptor``.
|
||
However, this PEP is already big enough
|
||
and this is left as a possible future improvement.
|
||
|
||
Slot wrappers for extension types like ``__init__`` or ``__eq__``
|
||
are quite different from normal methods.
|
||
They are also typically not called directly because you would normally
|
||
write ``foo[i]`` instead of ``foo.__getitem__(i)`` for example.
|
||
So these are left outside the scope of this PEP.
|
||
|
||
Python also has an ``instancemethod`` class, which was used in Python 2
|
||
for unbound methods.
|
||
It is not clear whether there is still a use case for it.
|
||
In any case, there is no reason to deal with it in this PEP.
|
||
|
||
**TODO**: should ``instancemethod`` be deprecated?
|
||
It doesn't seem used at all within CPython 3.7,
|
||
but maybe external packages use it?
|
||
|
||
Not treating METH_STATIC and METH_CLASS
|
||
---------------------------------------
|
||
|
||
Almost nothing in this PEP refers to the flags ``METH_STATIC`` and ``METH_CLASS``.
|
||
These flags are checked only by the `Automatic creation of built-in functions`_.
|
||
When a ``staticmethod``, ``classmethod`` or ``classmethod_descriptor``
|
||
is bound (i.e. ``__get__`` is called),
|
||
a ``base_function`` instance is created with ``m_self != NULL``.
|
||
For a ``classmethod``, this is obvious since ``m_self``
|
||
is the class that the method is bound to.
|
||
For a ``staticmethod``, one can take an arbitrary Python object for ``m_self``.
|
||
For backwards compatibility, we choose ``m_self = __objclass__`` for static methods
|
||
of extension types.
|
||
|
||
__self__ in base_function
|
||
-------------------------
|
||
|
||
It may look strange at first sight to add the ``__self__`` slot
|
||
in ``base_function`` as opposed to ``bound_method``.
|
||
We took this idea from the existing ``builtin_function_or_method`` class.
|
||
It allows us to have a single general implementation of ``__call__`` and ``__get__``
|
||
for the various function classes discussed in this PEP.
|
||
|
||
It also makes it easy to support existing built-in functions
|
||
which set ``__self__`` to the module (for example, ``sys.exit.__self__`` is ``sys``).
|
||
|
||
Subclassing
|
||
-----------
|
||
|
||
We disallow subclassing of ``builtin_function`` and ``bound_method``
|
||
to enable fast type checks for ``PyBuiltinFunction_Check`` and ``PyMethod_Check()``.
|
||
|
||
We allow subclassing of the other classes because there is no reason to disallow it.
|
||
For Python modules, the only relevant class to subclass is
|
||
``function`` because the others cannot be instantiated anyway.
|
||
|
||
Replacing tp_call: METH_ARG0_FUNCTION
|
||
-------------------------------------
|
||
|
||
The new flag ``METH_ARG0_FUNCTION`` is meant to support cases where
|
||
formerly a custom ``tp_call`` was used.
|
||
It would reduce the number of special fast paths in ``Python/ceval.c``
|
||
for calling objects:
|
||
instead of treating Python functions, built-in functions and methods,
|
||
there would only be a single check.
|
||
|
||
The signature of ``tp_call`` is essentially the signature
|
||
of ``PyBaseFunctionObject.m_ml.ml_meth`` with flags
|
||
``METH_VARARGS | METH_KEYWORDS | METH_ARG0_FUNCTION``.
|
||
Therefore, it should be easy to change existing ``tp_call`` slots
|
||
to use ``METH_ARG0_FUNCTION``.
|
||
There is one extra complication though: ``__self__`` must be handled manually.
|
||
|
||
User flags: METH_CUSTOM and METH_USRx
|
||
-------------------------------------
|
||
|
||
These flags are meant for applications that want to use
|
||
``tp_methods`` for an extension type or ``m_methods`` for a module
|
||
but that do not want the default built-in functions to be created.
|
||
Those applications would set ``METH_CUSTOM``.
|
||
The application is also free to use ``METH_USR0``, ..., ``METH_USR7``
|
||
for its own purposes,
|
||
for example to customize the creation of special function instances.
|
||
|
||
There is no obvious concrete use case,
|
||
but given that it costs essentially nothing to have these flags,
|
||
it seems like a good idea to allow it.
|
||
|
||
|
||
Backwards Compatibility
|
||
=======================
|
||
|
||
While designing this PEP, great care was taken to not break
|
||
backwards compatibility too much.
|
||
|
||
Python functions
|
||
----------------
|
||
|
||
For Python functions, essentially nothing changes.
|
||
The attributes that existed before still exist and Python functions
|
||
can be initialized, called and turned into methods as before.
|
||
|
||
The name ``function`` is kept for backwards compatibility.
|
||
While it might make sense to change the name to something more
|
||
specific like ``python_function``,
|
||
that would require a lot of annoying changes in documentation and testsuites.
|
||
|
||
Built-in functions of a module
|
||
------------------------------
|
||
|
||
Also for built-in functions, nothing changes.
|
||
We keep the old behaviour that such functions do not bind as methods.
|
||
This is a consequence of the fact that ``__self__`` is set to the module.
|
||
|
||
Built-in bound and unbound methods
|
||
----------------------------------
|
||
|
||
The types of built-in bound and unbound methods will change.
|
||
However, this does not affect calling such methods
|
||
because the protocol in ``base_function.__call__``
|
||
(in particular the handling of ``__objclass__`` and self slicing)
|
||
was specifically designed to be backwards compatible.
|
||
All attributes which existed before (like ``__objclass__`` and ``__self__``)
|
||
still exist.
|
||
|
||
New and changed classes
|
||
-----------------------
|
||
|
||
Tools which take various kinds of functions as input will need to deal
|
||
with the new function hieararchy and the possibility of custom
|
||
function classes.
|
||
The proposed changes to ``types`` and ``inspect``
|
||
are meant to minimize changes in behaviour.
|
||
However, it is unavoidable that some things change
|
||
and this can cause code to break.
|
||
In the Python standard library,
|
||
changes are needed in the ``doctest`` module because of this.
|
||
|
||
New attributes
|
||
--------------
|
||
|
||
Some objects get new special double-underscore attributes.
|
||
For example, ``__objclass__`` now appears on bound methods too
|
||
and all methods get a ``__func__`` attribute.
|
||
The fact that ``__self__`` is now a special read-only attribute
|
||
for Python functions caused trouble in [#bpo33265]_.
|
||
Generally, we expect that not much will break though.
|
||
|
||
method_descriptor and PyDescr_NewMethod
|
||
---------------------------------------
|
||
|
||
The classes ``method_descriptor`` and the constructor ``PyDescr_NewMethod``
|
||
are deprecated and no longer used by CPython itself.
|
||
They are kept for backwards compatibility though.
|
||
|
||
|
||
Reference Implementation
|
||
========================
|
||
|
||
Most of this PEP has been implemented for CPython at
|
||
https://github.com/jdemeyer/cpython/tree/pep575
|
||
|
||
There are four steps, corresponding to the commits on that branch.
|
||
After each step, CPython is in a mostly working state.
|
||
|
||
1. Add the ``base_function`` class and make it a subclass for ``builtin_function``.
|
||
This is by far the biggest step as the complete ``__call__`` protocol
|
||
is implemented in this step.
|
||
|
||
2. Rename ``method`` to ``bound_method`` and make it a subclass of ``base_function``.
|
||
Change unbound methods of extension types to be instances of ``builtin_function``
|
||
such that bound methods of extension types are also instances of ``bound_method``.
|
||
|
||
3. Implement ``defined_function`` and ``function``.
|
||
|
||
4. Changes to other parts of Python, such as the standard library and testsuite.
|
||
|
||
|
||
Appendix: current situation
|
||
===========================
|
||
|
||
**NOTE**:
|
||
This section is more useful during the draft period of the PEP,
|
||
so feel free to remove this once the PEP has been accepted.
|
||
|
||
For reference, we describe in detail the relevant existing classes in CPython 3.7.
|
||
|
||
Each of the classes involved is an "orphan" class
|
||
(no non-trivial subclasses nor superclasses).
|
||
|
||
builtin_function_or_method: built-in functions and bound methods
|
||
----------------------------------------------------------------
|
||
|
||
These are of type `PyCFunction_Type <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Objects/methodobject.c#L271>`_
|
||
with structure `PyCFunctionObject <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Include/methodobject.h#L102>`_::
|
||
|
||
typedef struct {
|
||
PyObject_HEAD
|
||
PyMethodDef *m_ml; /* Description of the C function to call */
|
||
PyObject *m_self; /* Passed as 'self' arg to the C func, can be NULL */
|
||
PyObject *m_module; /* The __module__ attribute, can be anything */
|
||
PyObject *m_weakreflist; /* List of weak references */
|
||
} PyCFunctionObject;
|
||
|
||
struct PyMethodDef {
|
||
const char *ml_name; /* The name of the built-in function/method */
|
||
PyCFunction ml_meth; /* The C function that implements it */
|
||
int ml_flags; /* Combination of METH_xxx flags, which mostly
|
||
describe the args expected by the C func */
|
||
const char *ml_doc; /* The __doc__ attribute, or NULL */
|
||
};
|
||
|
||
where ``PyCFunction`` is a C function pointer (there are various forms of this, the most basic
|
||
takes two arguments for ``self`` and ``*args``).
|
||
|
||
This class is used both for functions and bound methods:
|
||
for a method, the ``m_self`` slot points to the object::
|
||
|
||
>>> dict(foo=42).get
|
||
<built-in method get of dict object at 0x...>
|
||
>>> dict(foo=42).get.__self__
|
||
{'foo': 42}
|
||
|
||
In some cases, a function is considered a "method" of the module defining it::
|
||
|
||
>>> import os
|
||
>>> os.kill
|
||
<built-in function kill>
|
||
>>> os.kill.__self__
|
||
<module 'posix' (built-in)>
|
||
|
||
method_descriptor: built-in unbound methods
|
||
-------------------------------------------
|
||
|
||
These are of type `PyMethodDescr_Type <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Objects/descrobject.c#L538>`_
|
||
with structure `PyMethodDescrObject <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Include/descrobject.h#L53>`_::
|
||
|
||
typedef struct {
|
||
PyDescrObject d_common;
|
||
PyMethodDef *d_method;
|
||
} PyMethodDescrObject;
|
||
|
||
typedef struct {
|
||
PyObject_HEAD
|
||
PyTypeObject *d_type;
|
||
PyObject *d_name;
|
||
PyObject *d_qualname;
|
||
} PyDescrObject;
|
||
|
||
function: Python functions
|
||
--------------------------
|
||
|
||
These are of type `PyFunction_Type <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Objects/funcobject.c#L592>`_
|
||
with structure `PyFunctionObject <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Include/funcobject.h#L21>`_::
|
||
|
||
typedef struct {
|
||
PyObject_HEAD
|
||
PyObject *func_code; /* A code object, the __code__ attribute */
|
||
PyObject *func_globals; /* A dictionary (other mappings won't do) */
|
||
PyObject *func_defaults; /* NULL or a tuple */
|
||
PyObject *func_kwdefaults; /* NULL or a dict */
|
||
PyObject *func_closure; /* NULL or a tuple of cell objects */
|
||
PyObject *func_doc; /* The __doc__ attribute, can be anything */
|
||
PyObject *func_name; /* The __name__ attribute, a string object */
|
||
PyObject *func_dict; /* The __dict__ attribute, a dict or NULL */
|
||
PyObject *func_weakreflist; /* List of weak references */
|
||
PyObject *func_module; /* The __module__ attribute, can be anything */
|
||
PyObject *func_annotations; /* Annotations, a dict or NULL */
|
||
PyObject *func_qualname; /* The qualified name */
|
||
|
||
/* Invariant:
|
||
* func_closure contains the bindings for func_code->co_freevars, so
|
||
* PyTuple_Size(func_closure) == PyCode_GetNumFree(func_code)
|
||
* (func_closure may be NULL if PyCode_GetNumFree(func_code) == 0).
|
||
*/
|
||
} PyFunctionObject;
|
||
|
||
In Python 3, there is no "unbound method" class:
|
||
an unbound method is just a plain function.
|
||
|
||
method: Python bound methods
|
||
----------------------------
|
||
|
||
These are of type `PyMethod_Type <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Objects/classobject.c#L329>`_
|
||
with structure `PyMethodObject <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Include/classobject.h#L12>`_::
|
||
|
||
typedef struct {
|
||
PyObject_HEAD
|
||
PyObject *im_func; /* The callable object implementing the method */
|
||
PyObject *im_self; /* The instance it is bound to */
|
||
PyObject *im_weakreflist; /* List of weak references */
|
||
} PyMethodObject;
|
||
|
||
|
||
References
|
||
==========
|
||
|
||
.. [#cython] Cython (http://cython.org/)
|
||
|
||
.. [#bpo30071] Python bug 30071, Duck-typing inspect.isfunction() (https://bugs.python.org/issue30071)
|
||
|
||
.. [#bpo33261] Python bug 33261, inspect.isgeneratorfunction fails on hand-created methods
|
||
(https://bugs.python.org/issue33261 and https://github.com/python/cpython/pull/6448)
|
||
|
||
.. [#bpo33265] Python bug 33265, contextlib.ExitStack abuses __self__
|
||
(https://bugs.python.org/issue33265 and https://github.com/python/cpython/pull/6456)
|
||
|
||
.. [#ABI] PEP 384, Defining a Stable ABI, Löwis (https://www.python.org/dev/peps/pep-0384)
|
||
|
||
.. [#clinic] PEP 436, The Argument Clinic DSL, Hastings (https://www.python.org/dev/peps/pep-0436)
|
||
|
||
.. [#methoddoc] PyMethodDef documentation (https://docs.python.org/3.7/c-api/structures.html#c.PyMethodDef)
|
||
|
||
.. [#proposal] PEP proposal: unifying function/method classes (https://mail.python.org/pipermail/python-ideas/2018-March/049398.html)
|
||
|
||
Copyright
|
||
=========
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
|
||
..
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
coding: utf-8
|
||
End:
|