From 9012ec6e8c4264091214aed25e583d7947af9c0a Mon Sep 17 00:00:00 2001 From: jdemeyer Date: Mon, 2 Apr 2018 22:47:37 +0200 Subject: [PATCH] PEP 575: Unifying function/method classes (#605) --- pep-0575.rst | 901 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 901 insertions(+) create mode 100644 pep-0575.rst diff --git a/pep-0575.rst b/pep-0575.rst new file mode 100644 index 000000000..01854dcc1 --- /dev/null +++ b/pep-0575.rst @@ -0,0 +1,901 @@ +PEP: 575 +Title: Unifying function/method classes +Author: Jeroen Demeyer +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 27-Mar-2018 +Python-Version: 3.8 +Post-History: 31-Mar-2018 + + +Abstract +======== + +Reorganize the class hierarchy for functions and methods +with the goal of reducing the difference between +built-in functions (implemented in C) and Python functions. +Mainly, make built-in functions behave more like Python functions +without sacrificing performance. + +A new base class ``basefunction`` is introduced and the various function +classes, as well as ``method``, inherit from it. + +We also allow subclassing of some of these function classes. + +Motivation +========== + +Currently, CPython has two different function classes: +the first is Python functions, which is what you get +when defining a function with ``def`` or ``lambda``. +The second is built-in functions such as ``len``, ``isinstance`` or ``numpy.dot``. +These are implemented in C. + +These two classes are completely independent with different functionality. +In particular, it is currently not possible to implement a function efficiently in C +(only built-in functions can do that) +while still allowing introspection like ``inspect.signature`` or ``inspect.getsourcefile`` +(only Python functions can do that). +This is a problem for projects like Cython [#cython]_ that want to do exactly that. + +In Cython, this was worked around by inventing a new function class called ``cyfunction``. +Unfortunately, a new function class creates problems: +the ``inspect`` module does not recognize such functions as being functions [#bpo30071]_ +and the performance is worse +(CPython has specific optimizations for calling built-in functions). + +A second motivation is more generally making built-in functions and methods +behave more like Python functions and methods. +For example, Python unbound methods are just functions but +unbound methods of extension types (e.g. ``dict.get``) are a distinct class. +Bound methods of Python classes have a ``__func__`` attribute, +bound methods of extension types do not. + +New classes +=========== + +This is the new class hierarchy for functions and methods:: + + object + | + | + basefunction + / | \ + / | \ + / | generic_function + / | \ + builtin_function (*) | \ + | python_function + | + method (*) + +The two classes marked with (*) do *not* allow subclassing; +the others do. + +There is no difference between functions and unbound methods, +while bound methods are instances of ``method``. + +basefunction +------------ + +The class ``basefunction`` becomes a new base class for all function types. +It behaves like the existing ``builtin_function_or_method`` +with some differences: + +#. It acts as a descriptor implementing ``__get__`` to turn a function into a method + if there was no ``__self__`` attribute. + If the ``__self__`` attribute was already set, then this is a no-op: + the existing function is returned instead. + +#. A new read-only slot ``__objclass__``, represented in the C structure as ``m_selftype``. + If this attribute exists, it must be a class (it cannot be ``None``). + If so, the function must be called with ``self`` being an instance of that class. + This is meant to support unbound methods of extension types, replacing ``method_descriptor``. + +#. Argument Clinic [#clinic]_ is not supported. + +#. The field ``ml_doc`` and the attributes ``__doc__`` and ``__text_signature__`` + are gone. + +#. A new flag ``METH_CUSTOM`` for ``ml_flags`` which prevents automatic + generation of a ``builtin_function``, see `Automatic creation of built-in functions`_. + +#. A new flag ``METH_ARG0_FUNCTION`` for ``ml_flags``. + If this flag is set, the C function stored in ``ml_meth`` will be called with first argument + equal to the function object instead of ``__self__``. + +#. A new flag ``METH_ARG0_NO_SLICE`` for ``ml_flags``. + If this flag is *not* set, ``__objclass__`` is set and ``__self__`` is not set, + then the first positional argument is treated as ``__self__``. + For more details, see `Calling basefunction instances`_. + +#. A new flag ``METH_PYTHON`` for ``ml_flags``. + This flag indicates that this function should be treated as Python function. + Ideally, use of this flag should be avoided because it goes + against the duck typing philosophy. + It is still needed in a few places though, for example `Profiling`_. + +The goal of ``basefunction`` is that it supports all different ways +of calling functions and methods in just one structure. +For example, the new flag ``METH_ARG0_FUNCTION`` +will be used by the implementation of Python functions. + +It is not possible to directly create instances of ``basefunction`` +(``tp_new`` is ``NULL``). +However, it is legal for C code to manually create instances. + +These are the relevant C structures:: + + PyTypeObject PyBaseFunction_Type; + + typedef struct { + PyObject_HEAD + PyCFunctionDef *m_ml; /* Description of the C function to call */ + PyObject *m_self; /* __self__: anything, can be NULL; readonly */ + PyObject *m_module; /* __module__: anything */ + PyObject *m_weakreflist; /* List of weak references */ + PyObject *m_selftype; /* __objclass__: type object or NULL; readonly */ + } PyBaseFunctionObject; + + typedef struct { + const char *ml_name; /* The name of the built-in function/method */ + PyCFunction ml_meth; /* The C function that implements it */ + uint32_t ml_flags; /* Combination of METH_xxx flags, which mostly + describe the args expected by the C func */ + } PyCFunctionDef; + +Note that the type of ``ml_flags`` was changed from ``int`` to +``uint32_t`` (it makes a lot of sense to fix the number of bits). +Subclasses may extend ``PyCFunctionDef`` with extra fields. + +builtin_function +---------------- + +This is a copy of ``basefunction``, with the following differences: + +#. ``m_ml`` points to a ``PyMethodDef`` structure, + extending ``PyCFunctionDef`` with an additional ``ml_doc`` + field to implement ``__doc__`` and ``__text_signature__`` + as read-only attributes:: + + typedef struct { + const char *ml_name; + PyCFunction ml_meth; + uint32_t ml_flags; + const char *ml_doc; + } PyMethodDef; + +#. Argument Clinic [#clinic]_ is supported. + +The type object is ``PyTypeObject PyCFunction_Type`` +and we define ``PyCFunctionObject`` as alias of ``PyBaseFunctionObject``. + +generic_function +---------------- + +The class ``generic_function`` (a subclass of ``basefunction``) adds +support for various standard attributes which are used in ``inspect``. +This would be a good class to use for auto-generated C code, for example produced by Cython [#cython]_. + +The layout of the C structure is as follows:: + + PyTypeObject PyGenericFunction_Type; + + typedef struct { + PyBaseFunctionObject base; + PyObject *func_name; /* __name__: string */ + PyObject *func_qualname; /* __qualname__: string */ + PyObject *func_doc; /* __doc__: can be anything or NULL */ + PyObject *func_code; /* __code__: code or NULL */ + PyObject *func_defaults; /* __defaults__: tuple or NULL */ + PyObject *func_kwdefaults; /* __kwdefaults__: dict or NULL */ + PyObject *func_annotations; /* __annotations__: dict or NULL */ + PyObject *func_globals; /* __globals__: anything or NULL; readonly */ + PyObject *func_closure; /* __closure__: tuple of cell objects or NULL; readonly */ + PyObject *func_dict; /* __dict__: dict or NULL */ + } PyGenericFunctionObject; + +This class adds various slots like ``__doc__`` and ``__code__`` to access the C attributes. +The slot ``__name__`` returns ``func_name``. +When setting ``__name__``, also ``base.m_ml.ml_name`` is updated +with the UTF-8 encoded name. + +None of the attributes is required to be meaningful. +In particular, ``__code__`` may not be a working code object, +possibly only a few fields may be filled in. +And ``__defaults__`` is not required to be used for calling the function. + +Apart from adding these extra attributes, +``generic_function`` behaves exactly the same as ``basefunction``. + +python_function +--------------- + +This is the class meant for functions implemented in Python, +formerly known as ``function``. +Unlike the other function types, +instances of ``python_function`` can be created from Python code. + +The layout of the C structure is almost the same as ``generic_function``:: + + PyTypeObject PyFunction_Type; + + typedef struct { + PyBaseFunctionObject base; + PyObject *func_name; /* __name__: string */ + PyObject *func_qualname; /* __qualname__: string */ + PyObject *func_doc; /* __doc__: can be anything or NULL */ + PyObject *func_code; /* __code__: code or NULL */ + PyObject *func_defaults; /* __defaults__: tuple or NULL */ + PyObject *func_kwdefaults; /* __kwdefaults__: dict or NULL */ + PyObject *func_annotations; /* __annotations__: dict or NULL */ + PyObject *func_globals; /* __globals__: anything or NULL; readonly */ + PyObject *func_closure; /* __closure__: tuple of cell objects or NULL; readonly */ + PyObject *func_dict; /* __dict__: dict or NULL */ + PyCFunctionDef _ml; /* Storage for base.m_ml */ + } PyFunctionObject; + +The only difference is an ``_ml`` field +which reserves space to be used by ``base.m_ml``. +However, it is not required that ``base.m_ml`` points to ``_ml``. + +The constructor takes care of setting up ``base.m_ml``. +In particular, it sets the ``METH_PYTHON`` flag. + +method +------ + +The class ``method`` is used for all bound methods, +regardless of the class of the underlying function. +There is one extra attribute ``__func__`` pointing to that function. + +For methods, there is a complication because we want to allow +constructing a method from a arbitrary callable which +may not be an instance of ``basefunction``. +Therefore, in practice there are two kinds of methods: +for arbitrary callables, we use a single fixed ``PyCFunctionDef`` +structure with ``ml_name`` equal to ``"?"`` +and with the ``METH_ARG0_FUNCTION`` flag set. +The C function then calls ``__func__`` with the correct arguments. + +For methods which bind instances of ``basefunction`` +(more precisely, which have the ``Py_TPFLAGS_BASEFUNCTION`` flag set), +we instead use the ``PyCFunctionDef`` from the original function. +In this case, the ``__func__`` attribute is only used to implement various attributes +but not for calling the method. + +When constructing a new method from a ``basefunction``, +we check that the ``self`` object is an instance of ``__objclass__`` +(if such a class was specified) and raise a ``TypeError`` otherwise. + +The C structure is:: + + typedef struct { + PyBaseFunctionObject base; + PyObject *im_func; /* __func__: function implementing the method; readonly */ + } PyMethodObject; + + + +Calling basefunction instances +============================== + +We specify the implementation of ``__call__`` for instances of ``basefunction``. + +__objclass__ +------------ + +First of all, if the function has an ``__objclass__`` attribute but no +``__self__`` attribute (this is the case for unbound methods of extension types), +then the function must be called with at least one positional argument +and the first (typically called ``self``) must be an instance of ``__objclass__``. +If not, a ``TypeError`` is raised. + +Flags +----- + +For convenience, we define two new constants: +``METH_CALLSIGNATURE`` combines the flags from ``PyCFunctionDef.ml_flags`` +which specify the signature of the C function to be called. +It is equal to :: + + METH_NOARGS | METH_O | METH_VARARGS | METH_FASTCALL | METH_KEYWORDS + +Exactly one of the first four flags above must be set +and only ``METH_VARARGS`` and ``METH_FASTCALL`` may be combined with ``METH_KEYWORDS``. +Violating these rules is undefined behaviour. + +The second new constant is ``METH_CALLFLAGS``. +It combines all flags which influence how a function is called. +It is equal to :: + + METH_CALLSIGNATURE | METH_ARG0_FUNCTION | METH_ARG0_NO_SLICE + +Some of these flags are already documented [#methoddoc]_. +We explain the others below. + +METH_FASTCALL +------------- + +This is an existing but undocumented flag. +We suggest to officially support and document it. + +If the flag ``METH_FASTCALL`` is set without ``METH_KEYWORDS``, +then the ``ml_meth`` field is of type ``PyCFunctionFast`` +which takes the arguments ``(PyObject *arg0, PyObject *const *args, Py_ssize_t nargs)``. +Such a function takes only positional arguments and they are passed as plain C array +``args`` of length ``nargs``. + +If the flags ``METH_FASTCALL | METH_KEYWORDS`` are set, +then the ``ml_meth`` field is of type ``PyCFunctionFastWithKeywords`` +which takes the arguments ``(PyObject *arg0, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)``. +The positional arguments are passed as C array ``args`` of length ``nargs``. +The *values* of the keyword arguments follow in that array, +starting at position ``nargs``. +The *keys* (names) of the keyword arguments are passed as a ``tuple`` in ``kwnames``. +As an example, assume that 3 positional and 2 keyword arguments are given. +Then ``args`` is an array of length 3 + 2 = 5, ``nargs`` equals 3 and ``kwnames`` is a 2-tuple. + +METH_ARG0_FUNCTION +------------------ + +If this flag is set, then the first argument to the C function +is the function itself (the ``basefunction`` instance) instead of ``__self__``. +In this case, the C function should deal with ``__self__`` +by getting it from the function, for example using ``PyBaseFunction_GET_SELF``. + +METH_ARG0_NO_SLICE +------------------ + +If the function has a ``__objclass__`` attribute, no ``__self__`` +attribute and neither ``METH_ARG0_FUNCTION`` nor ``METH_ARG0_NO_SLICE`` are set, +then the first positional argument (which must exist because of ``__objclass__``) +is removed from ``*args`` and instead passed as first argument to the C function. +Effectively, the first positional argument is treated as ``__self__``. +This process is called "self slicing". +This does not affect keyword arguments. + +It is not allowed to combine the flags ``METH_ARG0_FUNCTION`` and ``METH_ARG0_NO_SLICE``. +That is not a problem because ``METH_ARG0_FUNCTION`` already disables self slicing. + + +Automatic creation of built-in functions +======================================== + +Python automatically generates instances of ``builtin_function`` +for extension types (using the ``PyTypeObject.tp_methods`` field) and modules +(using the ``PyModuleDef.m_methods`` field). +The arrays ``PyTypeObject.tp_methods`` and ``PyModuleDef.m_methods`` +must be arrays of ``PyMethodDef`` structures. + +If the ``METH_CUSTOM`` flag is set for an element of such an array, +then no ``builtin_function`` will be generated. +This allows an application to customize the creation of functions +in an extension type or module. +If ``METH_CUSTOM`` is set, then ``METH_STATIC`` and ``METH_CLASS`` are ignored. + +Built-in unbound methods +------------------------ + +The type of unbound methods changes from ``method_descriptor`` +to ``builtin_function``. +The object which appears as unbound method is the same object which +appears in the class ``__dict__``. +Python automatically sets the ``__objclass__`` attribute. + +Built-in functions of a module +------------------------------ + +For the case of functions of a module, +``__self__`` will be set to the module unless the flag ``METH_STATIC`` is set. + +An important consequence is that such functions by default +do not become methods when used as attribute +(``basefunction.__get__`` only does that if ``__self__`` was unset). +One could consider this a bug, but this was done for backwards compatibility reasons: +in an initial post on python-ideas [#proposal]_ the concensus was to keep this +misfeature of built-in functions. + +However, to allow this anyway for specific or newly implemented +built-in functions, the ``METH_STATIC`` flag prevents setting ``__self__``. +Previously, ``METH_STATIC`` was an error, so this is fullt backwards compatible. +Specifying ``METH_CLASS`` is still an error. + + +Further changes +=============== + +New type flag +------------- + +A new ``PyTypeObject`` flag (for ``tp_flags``) is added: +``Py_TPFLAGS_BASEFUNCTION`` to indicate that instances of this type are +functions which can be called as a ``basefunction``. +In other words, subclasses of ``basefunction`` +which follow the implementation from `Calling basefunction instances`_. + +This is different from flags like ``Py_TPFLAGS_LIST_SUBCLASS`` +because it indicates more than just a subclass: +it also indicates a default implementation of ``__call__``. +This flag is never inherited. +However, extension types can explicitly specify it if they +do not override ``__call__`` or if they override ``__call__`` in a compatible way. +The flag ``Py_TPFLAGS_BASEFUNCTION`` must never be set for a heap type +because that would not be safe (heap types can be changed dynamically). + +C API functions +--------------- + +We add and change some Python/C API functions: + +- ``int PyBaseFunction_Check(PyObject *op)``: return true if ``op`` + is an instance of a type with the ``Py_TPFLAGS_BASEFUNCTION`` set. + +- ``int PyCFunction_Check(PyObject *op)``: return true if ``PyBaseFunction_Check(op)`` + is True and the function ``op`` does not have the flag ``METH_PYTHON`` set. + +- ``int PyBuiltinFunction_Check(PyObject *op)``: return true if ``op`` + is an instance of ``builtin_function``. + +- ``int PyFunction_Check(PyObject *op)``: return true if ``op`` + is an instance of ``generic_function``. + +- ``PyObject* PyFunction_New(PyObject *code, PyObject *globals)``: + create a new instance of ``python_function``. + +- ``PyObject* PyFunction_NewWithQualName(PyObject *code, PyObject *globals)``: + create a new instance of ``python_function``. + +- For some existing ``PyCFunction_...`` and ``PyMethod_`` functions, + we define a new function ``PyBaseFunction_...`` + acting on ``basefunction`` instances. + For backwards compatibility, + the old functions are kept as aliases of the new functions. + +**TODO**: more functions may be added when implementing this PEP. +In particular, maybe there should be functions for creating instances of ``basefunction`` +or ``generic_function``. + +Changes to the types module +--------------------------- + +Two types are added: ``types.BaseFunctionType`` corresponding to +``basefunction`` and ``types.GenericFunctionType`` corresponding to +``generic_function``. + +Apart from that, no changes to the ``types`` module are made. +In particular, ``types.FunctionType`` refers to ``python_function``. +However, the actual types will change: +for example, ``types.BuiltinFunctionType`` will no longer be the same +as ``types.BuiltinMethodType``. + +Changes to the inspect module +----------------------------- + +``inspect.isbasefunction`` checks for an instance of ``basefunction``. + +``inspect.isfunction`` checks for an instance of ``generic_function``. + +``inspect.isbuiltin`` checks for an instance of ``builtin_function``. + +Profiling +--------- + +Currently, ``sys.setprofile`` supports ``c_call``, ``c_return`` and ``c_exception`` +events for built-in functions. +These events are generated when calling or returning from a built-in function. +By contrast, the ``call`` and ``return`` events are generated by the function itself. +So nothing needs to change for the ``call`` and ``return`` events. + +Since we no longer make a difference between C functions and Python functions, +we need to prevent the ``c_*`` events for Python functions. +This is done by not generating those events if the +``METH_PYTHON`` flag in ``ml_flags`` is set. + +User flags in PyCFunctionDef.ml_flags +---------------------------------------- + +8 consecutive bits in ``ml_flags`` are reserved for the "user", +meaning the person or program who implemented the function. +These are ``METH_USR0``, ..., ``METH_USR7``. +Python will ignore these flags. + +It should be clear that different users may use these flags +for different purposes, so users should only look at those flags in +functions that they implemented (for example, by looking for those flags +in the ``tp_methods`` array of an extension type). + + +Non-CPython implementations +=========================== + +For other implementations of Python apart from CPython, +only the classes ``basefunction``, ``method`` and ``python_function`` are required. +The latter two are the only classes which can be instantiated directly +from the Python interpreter. +We require ``basefunction`` for consistency but we put no requirements on it: +it is acceptable if this is just a copy of ``object``. +Support for the new ``__objclass__`` attribute is not required. +If there is no ``generic_function`` type, +then ``types.GenericFunctionType`` should be an alias of ``types.FunctionType``. + + +Rationale +========= + +Why not simply change existing classes? +--------------------------------------- + +One could try to solve the problem not by introducing a new ``basefunction`` +class and changing the class hierarchy, but by just changing existing classes. + +That might look like a simpler solution but it is not: +it would require introspection support for 3 distinct classes: +``function``, ``builtin_function_or_method`` and ``method_descriptor``. +In the current PEP, there is only a single class where introspection needs +to be implemented. +It is also not clear how this would interact with ``__text_signature__``. +Having two independent kinds of ``inspect.signature`` support on the same +class sounds like asking for problems. + +And this would not fix some of the other differences between built-in functions +and Python functions that were mentioned in the `Motivation`_. + +Why __text_signature__ is not a solution +---------------------------------------- + +Built-in functions have an attribute ``__text_signature__``, +which gives the signature of the function as plain text. +The default values are evaluated by ``ast.literal_eval``. +Because of this, it supports only a small number of standard Python classes +and not arbitrary Python objects. + +And even if ``__text_signature__`` would allow arbitrary signatures somehow, +that is only one piece of introspection: +it does not help with ``inspect.getsourcefile`` for example. + +generic_function versus python_function +--------------------------------------- + +The names ``generic_function`` and ``python_function`` +were chosen to be different from ``function`` +because none of the two classes ``generic_function``/``python_function`` +is an obvious candidate to receive the ``function`` name. +It also allows to use the word "function" informally without referring +to a specific class. + +In many places, a decision needs to be made whether the old ``function`` class +should be replaced by ``generic_function`` or ``python_function``. +This is done by thinking of the most likely use case: + +1. ``types.FunctionType`` refers to ``python_function`` because that + type might be used to construct instances using ``types.FunctionType(...)``. + +2. ``inspect.isfunction()`` refers to ``generic_function`` + because this is the class where introspection is supported. + +3. The C API functions ``PyFunction_New...`` + refer to ``python_function`` simply because one cannot create instances + of ``generic_function``. + +4. The C API functions ``PyFunction_Check`` and ``PyFunction_Get/Set...`` + refer to ``generic_function`` because all attributes exist for instances of ``generic_function``. + +Scope of this PEP: which classes are involved? +---------------------------------------------- + +The main motivation of this PEP is fixing function classes, +so we certainly want to unify the existing classes +``builtin_function_or_method`` and ``function``. + +Since built-in functions and methods have the same class, +it seems natural to include bound methods too. +And since there are no "unbound methods" for Python functions, +it makes sense to get rid of unbound methods for extension types. + +For now, no changes are made to the classes ``staticmethod``, +``classmethod`` and ``classmethod_descriptor``. +It would certainly make sense to put these in the ``basefunction`` +class hierarchy and unify ``classmethod`` and ``classmethod_descriptor``. +However, this PEP is already big enough +and this is left as a possible future improvement. + +Slot wrappers for extension types like ``__init__`` or ``__eq__`` +are quite different from normal methods. +They are also typically not called directly because you would normally +write ``foo[i]`` instead of ``foo.__getitem__(i)`` for example. +So these are left outside the scope of this PEP. + +Python also has an ``instancemethod`` class, which was used in Python 2 +for unbound methods. +It is not clear whether there is still a use case for it. +In any case, there is no reason to deal with it in this PEP. + +**TODO**: should ``instancemethod`` be deprecated? +It doesn't seem used at all within CPython 3.7, +but maybe external packages use it? + +__self__ in basefunction +------------------------ + +It may look strange at first sight to add the ``__self__`` slot +in ``basefunction`` as opposed to ``method``. +We took this idea from the existing ``builtin_function_or_method`` class. +It allows us to have a single general implementation of ``__call__`` +for the various function classes discussed in this PEP. +It also makes it easy to support existing built-in functions +which set ``__self__`` to the module (for example, ``sys.exit.__self__`` is ``sys``). + +Subclassing +----------- + +We disallow subclassing of ``builtin_function`` and ``method`` +to enable fast type checks for ``PyBuiltinFunction_Check`` and ``PyMethod_Check()``. + +We allow subclassing of the other classes because there is no reason to disallow it. +For Python modules, the only relevant class to subclass is +``python_function`` because the others cannot be instantiated anyway. + +Replacing tp_call: METH_ARG0_FUNCTION +------------------------------------- + +The new flag ``METH_ARG0_FUNCTION`` is meant to support cases where +formerly a custom ``tp_call`` was used. +It would reduce the number of special fast paths in ``Python/ceval.c`` +for calling objects: +instead of treating Python functions, built-in functions and methods, +there would only be a single check. + +The signature of ``tp_call`` is essentially the signature +of ``PyBaseFunctionObject.m_ml.ml_meth`` with flags +``METH_VARARGS | METH_KEYWORDS | METH_ARG0_FUNCTION``. +Therefore, it should be easy to change existing ``tp_call`` slots +to use ``METH_ARG0_FUNCTION``. +There is just one extra complication: ``__self__`` must be handled manually. +That is not hard though: it just means adapting that logic from ``method``. + +Self slicing: METH_ARG0_NO_SLICE +-------------------------------- + +We define "self slicing" to mean slicing off the ``self`` argument of a method +from the ``*args`` tuple when an unbound method is called. +This ``self`` argument is then passed as first argument to the C function. + +The specification of ``METH_ARG0_NO_SLICE`` may seem strange at first. +The negation is confusing, but it is done for backwards compatibility: +existing methods require self slicing but do not specify a flag for it. + +The requirement for ``__objclass__`` in order to use self slicing +makes sense because it guarantees that there is a ``self`` argument in the first place. + +Since ``METH_ARG0_FUNCTION`` is clearly incompatible with self slicing +(both use the first argument of the C function), +this PEP dictates that ``METH_ARG0_FUNCTION`` disables self slicing. +So one may wonder if there is actually a use case for ``METH_ARG0_NO_SLICE`` +without ``METH_ARG0_FUNCTION``. +If not, then one could simply unify those two flags in one flag +``METH_ARG0_FUNCTION``. + +However, a priori, the flag ``METH_ARG0_NO_SLICE`` is meaningful, +so we keep the two flags ``METH_ARG0_FUNCTION`` and ``METH_ARG0_NO_SLICE`` separate. + +**TODO**: this should be reconsidered after initial implementation +and testing of this PEP. + +User flags: METH_CUSTOM and METH_USRx +------------------------------------- + +These flags are meant for applications that want to use +``tp_methods`` for an extension type or ``m_methods`` for a module +but that do not want the default built-in functions to be created. +Those applications would set ``METH_CUSTOM``. +The application is also free to use ``METH_USR0``, ..., ``METH_USR7`` +for its own purposes, +for example to customize the creation of special function instances. + +There is no obvious concrete use case, +but given that it costs essentially nothing to have these flags, +it seems like a good idea to allow it. + + +Backwards Compatibility +======================= + +While designing this PEP, great care was taken to not break +backwards compatibility too much. + +Python functions +---------------- + +For Python functions, essentially nothing changes. +The attributes that existed before still exist and Python functions +can be initialized, called and turned into methods as before. + +Built-in functions of a module +------------------------------ + +Also for built-in functions, nothing changes. +We keep the old behaviour that such functions do not bind as methods. +This is a consequence of the fact that ``__self__`` is set to the module. + +Built-in bound and unbound methods +---------------------------------- + +The types of built-in bound and unbound methods will change. +However, this does not affect calling such methods +because the protocol in ``basefunction.__call__`` +(in particular the handling of ``__objclass__`` and self slicing) +was specifically designed to be backwards compatible. +All attributes which existed before (like ``__objclass__`` and ``__self__``) +still exist. + +New classes +----------- + +Tools which take various kinds of functions as input will need to deal +with the new function hieararchy and the possibility of custom +function classes. +If those tools use ``inspect`` properly, there should be few +backwards compatibility problems. + +New attributes +-------------- + +Some objects get new attributes. +For example, ``__objclass__`` now appears on bound methods too +and all methods get a ``__func__`` attribute. +We expect that this will not cause problems. + + +Reference Implementation +======================== + +After initial discussions of this PEP draft, +work will start on a reference implementation in CPython. + + +Appendix: current situation +=========================== + +**NOTE**: +This section is more useful during the draft period of the PEP, +so feel free to remove this once the PEP has been accepted. + +For reference, we describe in detail the relevant existing classes in CPython 3.7. + +There are a surprisingly large number of classes involved, +each of them is an "orphan" class (no non-trivial subclasses nor superclasses). + +builtin_function_or_method: built-in functions and bound methods +---------------------------------------------------------------- + +These are of type `PyCFunction_Type `_ +with structure `PyCFunctionObject `_:: + + typedef struct { + PyObject_HEAD + PyMethodDef *m_ml; /* Description of the C function to call */ + PyObject *m_self; /* Passed as 'self' arg to the C func, can be NULL */ + PyObject *m_module; /* The __module__ attribute, can be anything */ + PyObject *m_weakreflist; /* List of weak references */ + } PyCFunctionObject; + + struct PyMethodDef { + const char *ml_name; /* The name of the built-in function/method */ + PyCFunction ml_meth; /* The C function that implements it */ + int ml_flags; /* Combination of METH_xxx flags, which mostly + describe the args expected by the C func */ + const char *ml_doc; /* The __doc__ attribute, or NULL */ + }; + +where ``PyCFunction`` is a C function pointer (there are various forms of this, the most basic +takes two arguments for ``self`` and ``*args``). + +This class is used both for functions and bound methods: +for a method, the ``m_self`` slot points to the object:: + + >>> dict(foo=42).get + + >>> dict(foo=42).get.__self__ + {'foo': 42} + +In some cases, a function is considered a "method" of the module defining it:: + + >>> import os + >>> os.kill + + >>> os.kill.__self__ + + +method_descriptor: built-in unbound methods +------------------------------------------- + +These are of type `PyMethodDescr_Type `_ +with structure `PyMethodDescrObject `_:: + + typedef struct { + PyDescrObject d_common; + PyMethodDef *d_method; + } PyMethodDescrObject; + + typedef struct { + PyObject_HEAD + PyTypeObject *d_type; + PyObject *d_name; + PyObject *d_qualname; + } PyDescrObject; + +function: Python functions +-------------------------- + +These are of type `PyFunction_Type `_ +with structure `PyFunctionObject `_:: + + typedef struct { + PyObject_HEAD + PyObject *func_code; /* A code object, the __code__ attribute */ + PyObject *func_globals; /* A dictionary (other mappings won't do) */ + PyObject *func_defaults; /* NULL or a tuple */ + PyObject *func_kwdefaults; /* NULL or a dict */ + PyObject *func_closure; /* NULL or a tuple of cell objects */ + PyObject *func_doc; /* The __doc__ attribute, can be anything */ + PyObject *func_name; /* The __name__ attribute, a string object */ + PyObject *func_dict; /* The __dict__ attribute, a dict or NULL */ + PyObject *func_weakreflist; /* List of weak references */ + PyObject *func_module; /* The __module__ attribute, can be anything */ + PyObject *func_annotations; /* Annotations, a dict or NULL */ + PyObject *func_qualname; /* The qualified name */ + + /* Invariant: + * func_closure contains the bindings for func_code->co_freevars, so + * PyTuple_Size(func_closure) == PyCode_GetNumFree(func_code) + * (func_closure may be NULL if PyCode_GetNumFree(func_code) == 0). + */ + } PyFunctionObject; + +In Python 3, there is no "unbound method" class: +an unbound method is just a plain function. + +method: Python bound methods +---------------------------- + +These are of type `PyMethod_Type `_ +with structure `PyMethodObject `_:: + + typedef struct { + PyObject_HEAD + PyObject *im_func; /* The callable object implementing the method */ + PyObject *im_self; /* The instance it is bound to */ + PyObject *im_weakreflist; /* List of weak references */ + } PyMethodObject; + + +References +========== + +.. [#cython] Cython (http://cython.org/) + +.. [#bpo30071] Python bug 30071 (https://bugs.python.org/issue30071) + +.. [#clinic] PEP 436, The Argument Clinic DSL, Hastings (https://www.python.org/dev/peps/pep-0436) + +.. [#methoddoc] PyMethodDef documentation (https://docs.python.org/3.7/c-api/structures.html#c.PyMethodDef) + +.. [#proposal] PEP proposal: unifying function/method classes (https://mail.python.org/pipermail/python-ideas/2018-March/049398.html) + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: