PEP 580: new flag CCALL_DEFARG instead of CCALL_FUNCARG (#967)

pep-0580.rst

@@ -131,8 +131,8 @@ It must be a strictly positive integer.
 At that offset, a ``PyCCallRoot`` structure appears::
 
     typedef struct {
-        PyCCallDef *cr_ccall;
-        PyObject   *cr_self;     /* __self__ argument for methods */
+        const PyCCallDef *cr_ccall;
+        PyObject         *cr_self;  /* __self__ argument for methods */
     } PyCCallRoot;
 
 The ``PyCCallDef`` structure contains everything needed to describe how
@@ -255,32 +255,22 @@ signature flags:
 - ``CCALL_O``:
   ``cc_func(PyObject *self, PyObject *arg)``
 
-The flag ``CCALL_FUNCARG`` may be combined with any of these.
+The flag ``CCALL_DEFARG`` may be combined with any of these.
 If so, the C function takes an additional argument
-as first argument before ``self``.
-This argument is used to pass the function object (see NOTE 1 below).
+as first argument before ``self``,
+namely a const pointer to the ``PyCCallDef`` structure used for this call.
 For example, we have the following signature:
 
-- ``CCALL_FUNCARG | CCALL_VARARGS``:
-  ``cc_func(PyObject *func, PyObject *self, PyObject *args)``
+- ``CCALL_DEFARG | CCALL_VARARGS``:
+  ``cc_func(const PyCCallDef *def, PyObject *self, PyObject *args)``
 
-One exception is ``CCALL_FUNCARG | CCALL_NOARGS``:
+One exception is ``CCALL_DEFARG | CCALL_NOARGS``:
 the ``unused`` argument is dropped, so the signature becomes
 
-- ``CCALL_FUNCARG | CCALL_NOARGS``:
-  ``cc_func(PyObject *func, PyObject *self)``
+- ``CCALL_DEFARG | CCALL_NOARGS``:
+  ``cc_func(const PyCCallDef *def, PyObject *self)``
 
-**NOTE 1**: with "function object", we mean the ``self`` in ``__call__``.
-In the case of bound methods, it is currently unspecified
-whether this refers
-to the bound method or the original function (which is wrapped by the bound method).
-In the reference implementation, the bound method is passed.
-In the future, this may change to the wrapped function.
-Despite this ambiguity, the implementation of bound methods
-guarantees that ``PyCCall_CCALLDEF(func)``
-points to the ``PyCCallDef`` of the original function.
-
-**NOTE 2**: unlike the existing ``METH_...`` flags,
+**NOTE**: unlike the existing ``METH_...`` flags,
 the ``CCALL_...`` constants do not necessarily represent single bits.
 So checking ``if (cc_flags & CCALL_VARARGS)`` is not a valid way
 for checking the signature.
@@ -405,13 +395,13 @@ In other words, ``PyCCall_Check(func)`` must be true.
 
 Macros to access the ``PyCCallRoot`` and ``PyCCallDef`` structures:
 
-- ``PyCCallRoot *PyCCall_CCALLROOT(PyObject *func)``:
+- ``const PyCCallRoot *PyCCall_CCALLROOT(PyObject *func)``:
   pointer to the ``PyCCallRoot`` structure inside ``func``.
 
-- ``PyCCallDef *PyCCall_CCALLDEF(PyObject *func)``:
+- ``const PyCCallDef *PyCCall_CCALLDEF(PyObject *func)``:
   shorthand for ``PyCCall_CCALLROOT(func)->cr_ccall``.
 
-- ``PyCCallDef *PyCCall_FLAGS(PyObject *func)``:
+- ``uint32_t PyCCall_FLAGS(PyObject *func)``:
   shorthand for ``PyCCall_CCALLROOT(func)->cr_ccall->cc_flags``.
 
 - ``PyObject *PyCCall_SELF(PyOject *func)``:
@@ -452,7 +442,11 @@ the implementation becomes very similar, but they remain separate classes
 The ``PyCCallDef`` structure is simply stored
 as part of the object structure.
 Both classes use ``PyCFunctionObject`` as object structure.
-This is the new layout for both classes::
+This is the new layout for both classes:
+
+.. _PyCFunctionObject:
+
+::
 
     typedef struct {
         PyObject_HEAD
@@ -537,6 +531,8 @@ performance improvements are discussed:
 
 - https://mail.python.org/pipermail/python-dev/2018-July/154775.html
 
+- https://mail.python.org/pipermail/python-dev/2019-April/156954.html
+
 
 Stable ABI
 ==========
@@ -678,6 +674,31 @@ since it cannot distinguish at runtime between
 a function (without ``self`` argument) and a bound method (with ``self`` argument).
 The ``CCALL_SELFARG`` flag makes this difference explicit.
 
+Why CCALL_DEFARG?
+-----------------
+
+The flag ``CCALL_DEFARG`` gives the callee access to the ``PyCCallDef *``.
+There are various use cases for this:
+
+1. The callee can use the ``cc_parent`` field, which is useful for PEP 573.
+
+2. Applications are free to extend the ``PyCCallDef`` structure with user-defined
+   fields, which can then be accessed analogously.
+
+3. In the case where the ``PyCCallDef`` structure
+   is part of the object structure
+   (this is true for example for `PyCFunctionObject`_),
+   an appropriate offset can be subtracted from the ``PyCCallDef`` pointer
+   to get a pointer to the callable object defining that ``PyCCallDef``.
+
+An earlier version of this PEP defined a flag ``CCALL_FUNCARG``
+instead of ``CCALL_DEFARG`` which would pass the callable object
+to the callee.
+This had similar use cases, but there was some ambiguity for
+bound methods: should the "callable object" be the bound method
+object or the original function wrapped by the method?
+By passing the ``PyCCallDef *`` instead, this ambiguity is gone
+since the bound method uses the ``PyCCallDef *`` from the wrapped function.
 
 Replacing tp_print
 ------------------
@@ -717,6 +738,10 @@ where this PEP has been discussed:
 
 - https://mail.python.org/pipermail/python-dev/2018-October/155403.html
 
+- https://mail.python.org/pipermail/python-dev/2019-March/156853.html
+
+- https://mail.python.org/pipermail/python-dev/2019-March/156879.html
+
 
 Reference implementation
 ========================
@@ -724,6 +749,10 @@ Reference implementation
 The reference implementation can be found at
 https://github.com/jdemeyer/cpython/tree/pep580
 
+For an example of using the C call protocol,
+the following branch implements ``functools.lru_cache`` using PEP 580:
+https://github.com/jdemeyer/cpython/tree/lru580
+
 
 References
 ==========