python-peps/pep-0575.rst

PEP: 575
Title: Unifying function/method classes
Author: Jeroen Demeyer <J.Demeyer@UGent.be>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Mar-2018
Python-Version: 3.8
Post-History: 31-Mar-2018, 12-Apr-2018


Abstract
========

Reorganize the class hierarchy for functions and methods
with the goal of reducing the difference between
built-in functions (implemented in C) and Python functions.
Mainly, make built-in functions behave more like Python functions
without sacrificing performance.

A new base class ``base_function`` is introduced and the various function
classes, as well as ``method`` (renamed to ``bound_method``), inherit from it.

We also allow subclassing of some of these function classes.

Motivation
==========

Currently, CPython has two different function classes:
the first is Python functions, which is what you get
when defining a function with ``def`` or ``lambda``.
The second is built-in functions such as ``len``, ``isinstance`` or ``numpy.dot``.
These are implemented in C.

These two classes are completely independent with different functionality.
In particular, it is currently not possible to implement a function efficiently in C
(only built-in functions can do that)
while still allowing introspection like ``inspect.signature`` or ``inspect.getsourcefile``
(only Python functions can do that).
This is a problem for projects like Cython [#cython]_ that want to do exactly that.

In Cython, this was worked around by inventing a new function class called ``cyfunction``.
Unfortunately, a new function class creates problems:
the ``inspect`` module does not recognize such functions as being functions [#bpo30071]_
and the performance is worse
(CPython has specific optimizations for calling built-in functions).

A second motivation is more generally making built-in functions and methods
behave more like Python functions and methods.
For example, Python unbound methods are just functions but
unbound methods of extension types (e.g. ``dict.get``) are a distinct class.
Bound methods of Python classes have a ``__func__`` attribute,
bound methods of extension types do not.

New classes
===========

This is the new class hierarchy for functions and methods::

                       object
                          |
                          |
                   base_function
                  /       |     \
                 /        |      \
                /         |   defined_function
               /          |        \
    builtin_function (*)  |         \
                          |       function
                          |
                    bound_method (*)

The two classes marked with (*) do *not* allow subclassing;
the others do.

There is no difference between functions and unbound methods,
while bound methods are instances of ``bound_method``.

base_function
-------------

The class ``base_function`` becomes a new base class for all function types.
It behaves like the existing ``builtin_function_or_method``
with some differences:

#. It acts as a descriptor implementing ``__get__`` to turn a function into a method
   if ``m_self`` is ``NULL``.
   If ``m_self`` is not ``NULL``,
   then this is a no-op: the existing function is returned instead.

#. A new read-only slot ``__objclass__``, represented in the C structure as ``m_objclass``.
   If this attribute exists, it must be a class.
   If so, the function must be called with ``self`` being an instance of that class.
   This is mainly meant to support unbound methods of extension types,
   replacing ``method_descriptor``.
   Also ``__qualname__`` will use ``__objclass__`` as namespace
   (instead of ``m_self`` before).
   For methods of extension types, it is automatically assigned as the defining class
   (``__class__`` in plain Python).

#. Argument Clinic [#clinic]_ is not supported.

#. The field ``ml_doc`` and the attributes ``__doc__`` and ``__text_signature__``
   are gone.

#. A new flag ``METH_ARG0_FUNCTION`` for ``ml_flags``.
   If this flag is set, the C function stored in ``ml_meth`` will be called with first argument
   equal to the function object instead of ``m_self``.

#. A new flag ``METH_BINDING`` for ``ml_flags`` which only applies to
   functions of a module (not methods of a class).
   If this flag is set, then ``m_self`` will be set to ``NULL`` instead
   of the module.
   This allows the function to behave more like a Python function
   as it enables ``__get__``.
   This flag also enables `Self slicing`_.

#. A new flag ``METH_PYTHON`` for ``ml_flags``.
   This flag indicates that this function should be treated as Python function.
   Ideally, use of this flag should be avoided because it goes
   against the duck typing philosophy.
   It is still needed in a few places though, for example `Profiling`_.

#. A new flag ``METH_CUSTOM`` for ``ml_flags`` which prevents automatic
   generation of a ``builtin_function``, see `Automatic creation of built-in functions`_.

The goal of ``base_function`` is that it supports all different ways
of calling functions and methods in just one structure.
For example, the new flag ``METH_ARG0_FUNCTION``
will be used by the implementation of Python functions.

It is not possible to directly create instances of ``base_function``
(``tp_new`` is ``NULL``).
However, it is legal for C code to manually create instances.

These are the relevant C structures::

    PyTypeObject PyBaseFunction_Type;

    typedef struct {
        PyObject_HEAD
        PyCFunctionDef *m_ml;     /* Description of the C function to call */
        PyObject *m_self;         /* __self__: anything, can be NULL; readonly */
        PyObject *m_module;       /* __module__: anything (typically str) */
        PyTypeObject *m_objclass; /* __objclass__: type or NULL; readonly */
        PyObject *m_weakreflist;  /* List of weak references */
    } PyBaseFunctionObject;

    typedef struct {
        const char *ml_name;   /* The name of the built-in function/method */
        PyCFunction ml_meth;   /* The C function that implements it */
        int ml_flags;          /* Combination of METH_xxx flags, which mostly
                                  describe the args expected by the C func */
    } PyCFunctionDef;

Subclasses may extend ``PyCFunctionDef`` with extra fields.

The Python attribute ``__self__`` returns ``m_self``,
except if ``METH_STATIC`` is set.
In that case, ``None`` is returned.
If ``m_self`` is ``NULL``, then there is no ``__self__`` attribute at all.
For that reason, we write either ``m_self`` or ``__self__`` in this PEP
with slightly different meanings.

builtin_function
----------------

This is a copy of ``base_function``, with the following differences:

#. ``m_ml`` points to a ``PyMethodDef`` structure,
   extending ``PyCFunctionDef`` with an additional ``ml_doc``
   field to implement ``__doc__`` and ``__text_signature__``
   as read-only attributes::

    typedef struct {
        const char *ml_name;
        PyCFunction ml_meth;
        int ml_flags;
        const char *ml_doc;
    } PyMethodDef;

   Note that ``PyMethodDef`` is part of the Python Stable ABI [#ABI]_
   and it is used by most extension modules,
   so we absolutely cannot change this structure.

#. Argument Clinic [#clinic]_ is supported.

The type object is ``PyTypeObject PyCFunction_Type``
and we define ``PyCFunctionObject`` as alias of ``PyBaseFunctionObject``.

defined_function
----------------

The class ``defined_function`` (a subclass of ``base_function``) adds
support for various standard attributes which are used in ``inspect``.
This would be a good class to use for auto-generated C code, for example produced by Cython [#cython]_.

The layout of the C structure is as follows::

    PyTypeObject PyDefinedFunction_Type;

    typedef struct {
        PyBaseFunctionObject base;
        PyObject *func_code;        /* __code__: code */
        PyObject *func_globals;     /* __globals__: anything; readonly */
        PyObject *func_name;        /* __name__: string */
        PyObject *func_qualname;    /* __qualname__: string */
        PyObject *func_doc;         /* __doc__: can be anything or NULL */
        PyObject *func_defaults;    /* __defaults__: tuple or NULL */
        PyObject *func_kwdefaults;  /* __kwdefaults__: dict or NULL */
        PyObject *func_closure;     /* __closure__: tuple of cell objects or NULL; readonly */
        PyObject *func_annotations; /* __annotations__: dict or NULL */
        PyObject *func_dict;        /* __dict__: dict or NULL */
    } PyDefinedFunctionObject;

This class adds various slots like ``__doc__`` and ``__code__`` to access the C attributes.
The slot ``__name__`` returns ``func_name``.
When setting ``__name__``, also ``base.m_ml.ml_name`` is updated
with the UTF-8 encoded name.

None of the attributes is required to be meaningful.
In particular, ``__code__`` may not be a working code object,
possibly only a few fields may be filled in.
And ``__defaults__`` is not required to be used for calling the function.

Apart from adding these extra attributes,
``defined_function`` behaves exactly the same as ``base_function``.

**TODO**: maybe find a better name for ``defined_function``.
Other proposals: ``builtout_function`` (a function that is better built out; pun on builtin),
``generic_function`` (original proposal but conflicts with ``functools.singledispatch`` generic functions),
``user_function`` (defined by the user as opposed to CPython).

function
--------

This is the class meant for functions implemented in Python,
formerly known as ``function``.
Unlike the other function types,
instances of ``function`` can be created from Python code.
This is not changed, so we do not describe the details in this PEP.

The layout of the C structure is almost the same as ``defined_function``::

    PyTypeObject PyFunction_Type;

    typedef struct {
        PyBaseFunctionObject base;
        PyObject *func_code;        /* __code__: code */
        PyObject *func_globals;     /* __globals__: dict; readonly */
        PyObject *func_name;        /* __name__: string */
        PyObject *func_qualname;    /* __qualname__: string */
        PyObject *func_doc;         /* __doc__: can be anything or NULL */
        PyObject *func_defaults;    /* __defaults__: tuple or NULL */
        PyObject *func_kwdefaults;  /* __kwdefaults__: dict or NULL */
        PyObject *func_closure;     /* __closure__: tuple of cell objects or NULL; readonly */
        PyObject *func_annotations; /* __annotations__: dict or NULL */
        PyObject *func_dict;        /* __dict__: dict or NULL */
        PyCFunctionDef _ml;         /* Storage for base.m_ml */
    } PyFunctionObject;

The only difference is an ``_ml`` field
which reserves space to be used by ``base.m_ml``.

When constructing an instance of ``function`` from ``code`` and ``globals``,
an instance is created with ``base.m_ml = &_ml``,
``base.m_self = NULL`` and with the ``METH_PYTHON`` flag set.

To make subclassing easier, we also add a copy constructor:
if ``f`` is an instance of ``defined_function`` with the ``METH_PYTHON``
flag set, then ``types.FunctionType(f)`` copies ``f``.

bound_method
------------

The class ``bound_method`` is used for all bound methods,
regardless of the class of the underlying function.
It adds one new attribute on top of ``base_function``:
``__func__`` points to that function.

``bound_method`` replaces the old ``method`` class
which was used only for Python functions bound as method.

There is a complication because we want to allow
constructing a method from an arbitrary callable.
This may be an already-bound method or simply not an instance of ``base_function``.
Therefore, in practice there are two kinds of methods:

- For arbitrary callables, we use a single fixed ``PyCFunctionDef``
  structure with the ``METH_ARG0_FUNCTION`` flag set.
  The C function then calls ``__func__`` with the correct arguments.

- For methods which bind instances of ``base_function``
  (more precisely, which have the ``Py_TPFLAGS_BASEFUNCTION`` flag set)
  that allow self slicing,
  we instead use the ``PyCFunctionDef`` from the original function.
  In this case, the ``__func__`` attribute is only used to implement
  various attributes but not for calling the method.

When constructing a new method from a ``base_function``,
we check that the ``self`` object is an instance of ``__objclass__``
(if such a class was specified) and raise a ``TypeError`` otherwise.

The C structure is::

    PyTypeObject PyMethod_Type;

    typedef struct {
        PyBaseFunctionObject base;
        PyObject *im_func;  /* __func__: function implementing the method; readonly */
    } PyMethodObject;



Calling base_function instances
===============================

We specify the implementation of ``__call__`` for instances of ``base_function``.

__objclass__
------------

First of all, if the function has an ``__objclass__`` attribute but
``m_self`` is ``NULL`` (this is the case for unbound methods of extension types),
then the function must be called with at least one positional argument
and the first (typically called ``self``) must be an instance of ``__objclass__``.
If not, a ``TypeError`` is raised.

Flags
-----

For convenience, we define a new constant:
``METH_CALLSIGNATURE`` combines all flags from ``PyCFunctionDef.ml_flags``
which specify the signature of the C function to be called.
It is equal to ::

    METH_VARARGS | METH_FASTCALL | METH_NOARGS | METH_O | METH_KEYWORDS

Exactly one of the first four flags above must be set
and only ``METH_VARARGS`` and ``METH_FASTCALL`` may be combined with ``METH_KEYWORDS``.
Violating these rules is undefined behaviour.

There is one new flag which affects calling functions,
namely ``METH_ARG0_FUNCTION``.
Some flags are already documented in [#methoddoc]_.
We explain the other two shortly.

Self slicing
------------

If the function has ``m_self == NULL``
and the flag ``METH_ARG0_FUNCTION`` is not set,
then the first positional argument (if any)
is removed from ``*args`` and instead passed as first argument to the C function.
Effectively, the first positional argument is treated as ``__self__``.
This process is called "self slicing" and is meant to support unbound methods.
This does not affect keyword arguments.

METH_FASTCALL
-------------

This is an existing but undocumented flag.
We suggest to officially support and document it.

If the flag ``METH_FASTCALL`` is set without ``METH_KEYWORDS``,
then the ``ml_meth`` field is of type ``PyCFunctionFast``
which takes the arguments ``(PyObject *arg0, PyObject *const *args, Py_ssize_t nargs)``.
Such a function takes only positional arguments and they are passed as plain C array
``args`` of length ``nargs``.

If the flags ``METH_FASTCALL | METH_KEYWORDS`` are set,
then the ``ml_meth`` field is of type ``PyCFunctionFastWithKeywords``
which takes the arguments ``(PyObject *arg0, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)``.
The positional arguments are passed as C array ``args`` of length ``nargs``.
The *values* of the keyword arguments follow in that array,
starting at position ``nargs``.
The *keys* (names) of the keyword arguments are passed as a ``tuple`` in ``kwnames``.
As an example, assume that 3 positional and 2 keyword arguments are given.
Then ``args`` is an array of length 3 + 2 = 5, ``nargs`` equals 3 and ``kwnames`` is a 2-tuple.

METH_ARG0_FUNCTION
------------------

If this flag is set, then the first argument to the C function
is the function itself (the ``base_function`` instance) instead of ``m_self``.
In this case, the C function should deal with ``__self__``
by getting it from the function, for example using ``PyBaseFunction_GET_SELF``.


Automatic creation of built-in functions
========================================

Python automatically generates instances of ``builtin_function``
for extension types (using the ``PyTypeObject.tp_methods`` field) and modules
(using the ``PyModuleDef.m_methods`` field).
The arrays ``PyTypeObject.tp_methods`` and ``PyModuleDef.m_methods``
must be arrays of ``PyMethodDef`` structures.

If the ``METH_CUSTOM`` flag is set for an element of such an array,
then no ``builtin_function`` will be generated.
This allows an application to customize the creation of functions
in an extension type or module.
If ``METH_CUSTOM`` is set, then ``METH_STATIC`` and ``METH_CLASS`` are ignored.

Built-in unbound methods
------------------------

The type of unbound methods changes from ``method_descriptor``
to ``builtin_function``.
The object which appears as unbound method is the same object which
appears in the class ``__dict__``.
Python automatically sets the ``__objclass__`` attribute.

Built-in functions of a module
------------------------------

For the case of functions of a module,
``__self__`` will be set to the module unless the flag ``METH_BINDING`` is set.

An important consequence is that such functions by default
do not become methods when used as attribute
(``base_function.__get__`` only does that if ``m_self`` was ``NULL``).
One could consider this a bug, but this was done for backwards compatibility reasons:
in an initial post on python-ideas [#proposal]_ the concensus was to keep this
misfeature of built-in functions.

However, to allow this anyway for specific or newly implemented
built-in functions, the ``METH_BINDING`` flag prevents setting ``__self__``.


Further changes
===============

New type flag
-------------

A new ``PyTypeObject`` flag (for ``tp_flags``) is added:
``Py_TPFLAGS_BASEFUNCTION`` to indicate that instances of this type are
functions which can be called as a ``base_function``.
In other words, subclasses of ``base_function``
which follow the implementation from `Calling base_function instances`_.

This is different from flags like ``Py_TPFLAGS_LIST_SUBCLASS``
because it indicates more than just a subclass:
it also indicates a default implementation of ``__call__``.
This flag is never inherited.
However, extension types can explicitly specify it if they
do not override ``__call__`` or if they override ``__call__`` in a compatible way.
The flag ``Py_TPFLAGS_BASEFUNCTION`` must never be set for a heap type
because that would not be safe (heap types can be changed dynamically).

C API functions
---------------

We list some relevant Python/C API macros and functions.
Some of these are existing (possibly changed) functions, some are new:

- ``int PyBaseFunction_CheckFast(PyObject *op)``: return true if ``op``
  is an instance of a class with the ``Py_TPFLAGS_BASEFUNCTION`` set.
  This is the function that you need to use to determine
  whether you can safely access the ``base_function`` internals.

- ``int PyBaseFunction_Check(PyObject *op)``: return true if ``op``
  is an instance of ``base_function``.

- ``PyObject *PyBaseFunction_New(PyTypeObject *cls, PyCFunctionDef *ml, PyObject *self, PyObject *module, PyTypeObject *objclass)``:
  create a new instance of ``cls`` (which must be a subclass of ``base_function``)
  from the given data.

- ``int PyCFunction_Check(PyObject *op)``: return true if ``op``
  is an instance of ``builtin_function``.

- ``int PyCFunction_NewEx(PyMethodDef* ml, PyObject *self, PyObject* module)``:
  create a new instance of ``builtin_function``.
  As special case, if ``self`` is ``NULL``,
  then set ``self = Py_None`` instead (for backwards compatibility).

- For many existing ``PyCFunction_...`` and ``PyMethod_`` functions,
  we define a new function ``PyBaseFunction_...``
  acting on ``base_function`` instances.
  For backwards compatibility,
  the old functions are kept as aliases of the new functions.

- ``int PyFunction_Check(PyObject *op)``: return true if ``op``
  is an instance of ``defined_function``.

- ``PyObject *PyFunction_NewPython(PyTypeObject *cls, PyObject *code, PyObject *globals, PyObject *name, PyObject *qualname)``:
  create a new instance of ``cls`` (which must be a sublass of ``function``)
  from the given data.

- ``PyObject *PyFunction_New(PyObject *code, PyObject *globals)``:
  create a new instance of ``function``.

- ``PyObject *PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname)``:
  create a new instance of ``function``.

- ``PyObject *PyFunction_Copy(PyTypeObject *cls, PyObject *func)``:
  create a new instance of ``cls`` (which must be a sublass of ``function``)
  by copying a given ``defined_function``.

- All other existing ``PyFunction_...`` functions now act on ``defined_function``
  instances (instead of ``function``).

Changes to the types module
---------------------------

Two types are added: ``types.BaseFunctionType`` corresponding to
``base_function`` and ``types.DefinedFunctionType`` corresponding to
``defined_function``.

Apart from that, no changes to the ``types`` module are made.
In particular, ``types.FunctionType`` refers to ``function``.
However, the actual types will change:
for example, ``types.BuiltinFunctionType`` will no longer be the same
as ``types.BuiltinMethodType``.

Changes to the inspect module
-----------------------------

``inspect.isbasefunction`` checks for an instance of ``base_function``.

``inspect.isfunction`` checks for an instance of ``defined_function``.

``inspect.isbuiltin`` checks for an instance of ``builtin_function``.

``inspect.isroutine`` checks ``isbasefunction`` or ``ismethoddescriptor``.

Note that bpo-33261 [#bpo33261]_ should be fixed first.

Profiling
---------

Currently, ``sys.setprofile`` supports ``c_call``, ``c_return`` and ``c_exception``
events for built-in functions.
These events are generated when calling or returning from a built-in function.
By contrast, the ``call`` and ``return`` events are generated by the function itself.
So nothing needs to change for the ``call`` and ``return`` events.

Since we no longer make a difference between C functions and Python functions,
we need to prevent the ``c_*`` events for Python functions.
This is done by not generating those events if the
``METH_PYTHON`` flag in ``ml_flags`` is set.

User flags in PyCFunctionDef.ml_flags: METH_USRx
------------------------------------------------

8 consecutive bits in ``ml_flags`` are reserved for the "user",
meaning the person or program who implemented the function.
These are ``METH_USR0``, ..., ``METH_USR7``.
Python will ignore these flags.

It should be clear that different users may use these flags
for different purposes, so users should only look at those flags in
functions that they implemented (for example, by looking for those flags
in the ``tp_methods`` array of an extension type).


Non-CPython implementations
===========================

For other implementations of Python apart from CPython,
only the classes ``base_function``, ``bound_method`` and ``function`` are required.
The latter two are the only classes which can be instantiated directly
from the Python interpreter.
We require ``base_function`` for consistency but we put no requirements on it:
it is acceptable if this is just a copy of ``object``.
Support for the new ``__objclass__`` attribute is not required.
If there is no ``defined_function`` type,
then ``types.DefinedFunctionType`` should be an alias of ``types.FunctionType``.


Rationale
=========

Why not simply change existing classes?
---------------------------------------

One could try to solve the problem not by introducing a new ``base_function``
class and changing the class hierarchy, but by just changing existing classes.

That might look like a simpler solution but it is not:
it would require introspection support for 3 distinct classes:
``function``, ``builtin_function_or_method`` and ``method_descriptor``.
In the current PEP, there is only a single class where introspection needs
to be implemented.
It is also not clear how this would interact with ``__text_signature__``.
Having two independent kinds of ``inspect.signature`` support on the same
class sounds like asking for problems.

And this would not fix some of the other differences between built-in functions
and Python functions that were mentioned in the `Motivation`_.

Why __text_signature__ is not a solution
----------------------------------------

Built-in functions have an attribute ``__text_signature__``,
which gives the signature of the function as plain text.
The default values are evaluated by ``ast.literal_eval``.
Because of this, it supports only a small number of standard Python classes
and not arbitrary Python objects.

And even if ``__text_signature__`` would allow arbitrary signatures somehow,
that is only one piece of introspection:
it does not help with ``inspect.getsourcefile`` for example.

defined_function versus function
--------------------------------

In many places, a decision needs to be made whether the old ``function`` class
should be replaced by ``defined_function`` or the new ``function`` class.
This is done by thinking of the most likely use case:

1. ``types.FunctionType`` refers to ``function`` because that
   type might be used to construct instances using ``types.FunctionType(...)``.

2. ``inspect.isfunction()`` refers to ``defined_function``
   because this is the class where introspection is supported.

3. The C API functions ``PyFunction_New...``
   refer to ``function`` simply because one cannot create instances
   of ``defined_function``.

4. The C API functions ``PyFunction_Check`` and ``PyFunction_Get/Set...``
   refer to ``defined_function`` because all attributes exist for instances of ``defined_function``.

Scope of this PEP: which classes are involved?
----------------------------------------------

The main motivation of this PEP is fixing function classes,
so we certainly want to unify the existing classes
``builtin_function_or_method`` and ``function``.

Since built-in functions and methods have the same class,
it seems natural to include bound methods too.
And since there are no "unbound methods" for Python functions,
it makes sense to get rid of unbound methods for extension types.

For now, no changes are made to the classes ``staticmethod``,
``classmethod`` and ``classmethod_descriptor``.
It would certainly make sense to put these in the ``base_function``
class hierarchy and unify ``classmethod`` and ``classmethod_descriptor``.
However, this PEP is already big enough
and this is left as a possible future improvement.

Slot wrappers for extension types like ``__init__`` or ``__eq__``
are quite different from normal methods.
They are also typically not called directly because you would normally
write ``foo[i]`` instead of ``foo.__getitem__(i)`` for example.
So these are left outside the scope of this PEP.

Python also has an ``instancemethod`` class, which was used in Python 2
for unbound methods.
It is not clear whether there is still a use case for it.
In any case, there is no reason to deal with it in this PEP.

**TODO**: should ``instancemethod`` be deprecated?
It doesn't seem used at all within CPython 3.7,
but maybe external packages use it?

Not treating METH_STATIC and METH_CLASS
---------------------------------------

Almost nothing in this PEP refers to the flags ``METH_STATIC`` and ``METH_CLASS``.
These flags are checked only by the `Automatic creation of built-in functions`_.
When a ``staticmethod``, ``classmethod`` or ``classmethod_descriptor``
is bound (i.e. ``__get__`` is called),
a ``base_function`` instance is created with ``m_self != NULL``.
For a ``classmethod``, this is obvious since ``m_self``
is the class that the method is bound to.
For a ``staticmethod``, one can take an arbitrary Python object for ``m_self``.
For backwards compatibility, we choose ``m_self = __objclass__`` for static methods
of extension types.

__self__ in base_function
-------------------------

It may look strange at first sight to add the ``__self__`` slot
in ``base_function`` as opposed to ``bound_method``.
We took this idea from the existing ``builtin_function_or_method`` class.
It allows us to have a single general implementation of ``__call__`` and ``__get__``
for the various function classes discussed in this PEP.

It also makes it easy to support existing built-in functions
which set ``__self__`` to the module (for example, ``sys.exit.__self__`` is ``sys``).

Two implementations of __doc__
------------------------------

``base_function`` does not support function docstrings.
Instead, the classes ``builtin_function`` and ``defined_function``
each have their own way of dealing with docstrings
(and ``bound_method`` just takes the ``__doc__`` from the wrapped function).

For ``builtin_function``, the docstring is stored (together with the text signature)
as C string in the read-only ``ml_doc`` field of a ``PyMethodDef``.
For ``defined_function``, the docstring is stored as a writable Python object
and it does not actually need to be a string.
This is done like this for backwards compatibility and because
it looks hard to unify these two very different ways of dealing with ``__doc__``.

Subclassing
-----------

We disallow subclassing of ``builtin_function`` and ``bound_method``
to enable fast type checks for ``PyBuiltinFunction_Check`` and ``PyMethod_Check()``.

We allow subclassing of the other classes because there is no reason to disallow it.
For Python modules, the only relevant class to subclass is
``function`` because the others cannot be instantiated anyway.

Replacing tp_call: METH_ARG0_FUNCTION
-------------------------------------

The new flag ``METH_ARG0_FUNCTION`` is meant to support cases where
formerly a custom ``tp_call`` was used.
It would reduce the number of special fast paths in ``Python/ceval.c``
for calling objects:
instead of treating Python functions, built-in functions and methods,
there would only be a single check.

The signature of ``tp_call`` is essentially the signature
of ``PyBaseFunctionObject.m_ml.ml_meth`` with flags
``METH_VARARGS | METH_KEYWORDS | METH_ARG0_FUNCTION``.
Therefore, it should be easy to change existing ``tp_call`` slots
to use ``METH_ARG0_FUNCTION``.
There is one extra complication though: ``__self__`` must be handled manually.

User flags: METH_CUSTOM and METH_USRx
-------------------------------------

These flags are meant for applications that want to use
``tp_methods`` for an extension type or ``m_methods`` for a module
but that do not want the default built-in functions to be created.
Those applications would set ``METH_CUSTOM``.
The application is also free to use ``METH_USR0``, ..., ``METH_USR7``
for its own purposes,
for example to customize the creation of special function instances.

There is no immediate concrete use case,
but we expect that tools which auto-generate functions or extension types
may want to define custom flags.
Given that it costs essentially nothing to have these flags,
it seems like a good idea to allow it.


Backwards compatibility
=======================

While designing this PEP, great care was taken to not break
backwards compatibility too much.
In particular, Python code not using ``inspect`` or type checks
should not be affected by this PEP.
For example, ``staticmethod``, ``functools.partial`` or ``operator.methodcaller``
do not need to change at all.

Changes to types and inspect
----------------------------

The proposed changes to ``types`` and ``inspect``
are meant to minimize changes in behaviour.
However, it is unavoidable that some things change
and this can cause code which uses ``types`` or ``inspect`` to break.
In the Python standard library for example,
changes are needed in the ``doctest`` module because of this.

Also, tools which take various kinds of functions as input will need to deal
with the new function hieararchy and the possibility of custom
function classes.

Python functions
----------------

For Python functions, essentially nothing changes.
The attributes that existed before still exist and Python functions
can be initialized, called and turned into methods as before.

The name ``function`` is kept for backwards compatibility.
While it might make sense to change the name to something more
specific like ``python_function``,
that would require a lot of annoying changes in documentation and testsuites.

Built-in functions of a module
------------------------------

Also for built-in functions, nothing changes.
We keep the old behaviour that such functions do not bind as methods.
This is a consequence of the fact that ``__self__`` is set to the module.

Built-in bound and unbound methods
----------------------------------

The types of built-in bound and unbound methods will change.
However, this does not affect calling such methods
because the protocol in ``base_function.__call__``
(in particular the handling of ``__objclass__`` and self slicing)
was specifically designed to be backwards compatible.
All attributes which existed before (like ``__objclass__`` and ``__self__``)
still exist.

New attributes
--------------

Some objects get new special double-underscore attributes.
For example, ``__objclass__`` now appears on bound methods too
and all methods get a ``__func__`` attribute.
The fact that ``__self__`` is now a special read-only attribute
for Python functions caused trouble in [#bpo33265]_.
Generally, we expect that not much will break though.

method_descriptor and PyDescr_NewMethod
---------------------------------------

The classes ``method_descriptor`` and the constructor ``PyDescr_NewMethod``
are deprecated and no longer used by CPython itself.
For now, they are kept for backwards compatibility.


Two-phase Implementation
========================

**TODO**: this section is optional. When accepting this PEP, it should
be decided whether to apply this two-phase implementation or not.

As mentioned above, the `changes to types and inspect`_ can break some
existing code.
In order to further minimize breakage, this PEP could be implemented
in two phases.

Phase one: duplicate classes
----------------------------

Implement this PEP but duplicate the classes ``bound_method``
and ``builtin_function``.
Add a new class ``builtin_method`` which is an exact copy of ``builtin_function``
add a new class ``bound_builtin_method`` which is an exact copy
of ``bound_method`` (in both cases, only the name would differ).

The class ``builtin_method`` will be used for unbound methods
of extension types.
It should be seen as continuation of the existing class
``method-descriptor``.
This ensures 100% backwards compatibility for these objects
(except for added attributes and maybe the ``repr()``).

The same would be done for bound methods of extension types:
these will be instances of ``bound_builtin_method``.
This ensures full backwards compatibility, except for code
assuming that ``types.BuiltinFunctionType`` is the same as ``types.BuiltinMethodType``.

For ``inspect``, we keep but deprecate the functions
``isbuiltin``, ``ismethod`` and ``ismethoddescriptor``.
To replace these, new functions ``isbuiltinfunction``, ``isboundmethod``
and ``isgetdescriptor`` (other possible names: ``isreaddescriptor`` or ``isdescriptor``)
are added.
The function ``isbuiltinfunction`` checks for instances of ``builtin_function``
and ``builtin_method``.
``isboundmethod`` checks for both ``bound_method`` and ``bound_builtin_method``.
And ``isgetdescriptor`` checks for non-data descriptors
which are not instances of ``base_function``.

Since showing an actual ``DeprecationWarning`` would affect a lot
of correctly-functioning code,
the deprecations would only appear in the documentation.
Another reason is that it is hard to show warnings for calling ``isinstance(x, t)``
(but it could be done using ``__instancecheck__`` hacking)
and impossible for ``type(x) is t``.

Phase two
---------

Phase two is what is actually described in the rest of this PEP:
the duplicate classes would be merged and the ``inspect`` functions
adjusted accordingly.


Reference Implementation
========================

Most of this PEP has been implemented for CPython at
https://github.com/jdemeyer/cpython/tree/pep575

There are four steps, corresponding to the commits on that branch.
After each step, CPython is in a mostly working state.

1. Add the ``base_function`` class and make it a subclass for ``builtin_function``.
   This is by far the biggest step as the complete ``__call__`` protocol
   is implemented in this step.

2. Rename ``method`` to ``bound_method`` and make it a subclass of ``base_function``.
   Change unbound methods of extension types to be instances of ``builtin_function``
   such that bound methods of extension types are also instances of ``bound_method``.

3. Implement ``defined_function`` and ``function``.

4. Changes to other parts of Python, such as the standard library and testsuite.


Appendix: current situation
===========================

**NOTE**:
This section is more useful during the draft period of the PEP,
so feel free to remove this once the PEP has been accepted.

For reference, we describe in detail the relevant existing classes in CPython 3.7.

Each of the classes involved is an "orphan" class
(no non-trivial subclasses nor superclasses).

builtin_function_or_method: built-in functions and bound methods
----------------------------------------------------------------

These are of type `PyCFunction_Type <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Objects/methodobject.c#L271>`_
with structure `PyCFunctionObject <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Include/methodobject.h#L102>`_::

    typedef struct {
        PyObject_HEAD
        PyMethodDef *m_ml; /* Description of the C function to call */
        PyObject    *m_self; /* Passed as 'self' arg to the C func, can be NULL */
        PyObject    *m_module; /* The __module__ attribute, can be anything */
        PyObject    *m_weakreflist; /* List of weak references */
    } PyCFunctionObject;

    struct PyMethodDef {
        const char  *ml_name;   /* The name of the built-in function/method */
        PyCFunction ml_meth;    /* The C function that implements it */
        int         ml_flags;   /* Combination of METH_xxx flags, which mostly
                                   describe the args expected by the C func */
        const char  *ml_doc;    /* The __doc__ attribute, or NULL */
    };

where ``PyCFunction`` is a C function pointer (there are various forms of this, the most basic
takes two arguments for ``self`` and ``*args``).

This class is used both for functions and bound methods:
for a method, the ``m_self`` slot points to the object::

    >>> dict(foo=42).get
    <built-in method get of dict object at 0x...>
    >>> dict(foo=42).get.__self__
    {'foo': 42}

In some cases, a function is considered a "method" of the module defining it::

    >>> import os
    >>> os.kill
    <built-in function kill>
    >>> os.kill.__self__
    <module 'posix' (built-in)>

method_descriptor: built-in unbound methods
-------------------------------------------

These are of type `PyMethodDescr_Type <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Objects/descrobject.c#L538>`_
with structure `PyMethodDescrObject <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Include/descrobject.h#L53>`_::

    typedef struct {
        PyDescrObject d_common;
        PyMethodDef *d_method;
    } PyMethodDescrObject;

    typedef struct {
        PyObject_HEAD
        PyTypeObject *d_type;
        PyObject *d_name;
        PyObject *d_qualname;
    } PyDescrObject;

function: Python functions
--------------------------

These are of type `PyFunction_Type <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Objects/funcobject.c#L592>`_
with structure `PyFunctionObject <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Include/funcobject.h#L21>`_::

    typedef struct {
        PyObject_HEAD
        PyObject *func_code;        /* A code object, the __code__ attribute */
        PyObject *func_globals;     /* A dictionary (other mappings won't do) */
        PyObject *func_defaults;    /* NULL or a tuple */
        PyObject *func_kwdefaults;  /* NULL or a dict */
        PyObject *func_closure;     /* NULL or a tuple of cell objects */
        PyObject *func_doc;         /* The __doc__ attribute, can be anything */
        PyObject *func_name;        /* The __name__ attribute, a string object */
        PyObject *func_dict;        /* The __dict__ attribute, a dict or NULL */
        PyObject *func_weakreflist; /* List of weak references */
        PyObject *func_module;      /* The __module__ attribute, can be anything */
        PyObject *func_annotations; /* Annotations, a dict or NULL */
        PyObject *func_qualname;    /* The qualified name */

        /* Invariant:
         *     func_closure contains the bindings for func_code->co_freevars, so
         *     PyTuple_Size(func_closure) == PyCode_GetNumFree(func_code)
         *     (func_closure may be NULL if PyCode_GetNumFree(func_code) == 0).
         */
    } PyFunctionObject;

In Python 3, there is no "unbound method" class:
an unbound method is just a plain function.

method: Python bound methods
----------------------------

These are of type `PyMethod_Type <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Objects/classobject.c#L329>`_
with structure `PyMethodObject <https://github.com/python/cpython/blob/2cb4661707818cfd92556e7fdf9068a993577002/Include/classobject.h#L12>`_::

    typedef struct {
        PyObject_HEAD
        PyObject *im_func;   /* The callable object implementing the method */
        PyObject *im_self;   /* The instance it is bound to */
        PyObject *im_weakreflist; /* List of weak references */
    } PyMethodObject;


References
==========

.. [#cython] Cython (http://cython.org/)

.. [#bpo30071] Python bug 30071, Duck-typing inspect.isfunction() (https://bugs.python.org/issue30071)

.. [#bpo33261] Python bug 33261, inspect.isgeneratorfunction fails on hand-created methods
   (https://bugs.python.org/issue33261 and https://github.com/python/cpython/pull/6448)

.. [#bpo33265] Python bug 33265, contextlib.ExitStack abuses __self__
   (https://bugs.python.org/issue33265 and https://github.com/python/cpython/pull/6456)

.. [#ABI] PEP 384, Defining a Stable ABI, Löwis (https://www.python.org/dev/peps/pep-0384)

.. [#clinic] PEP 436, The Argument Clinic DSL, Hastings (https://www.python.org/dev/peps/pep-0436)

.. [#methoddoc] PyMethodDef documentation (https://docs.python.org/3.7/c-api/structures.html#c.PyMethodDef)

.. [#proposal] PEP proposal: unifying function/method classes (https://mail.python.org/pipermail/python-ideas/2018-March/049398.html)

Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: