Update PEP 575 (#619)

pep-0575.rst

@@ -6,7 +6,7 @@ Type: Standards Track
 Content-Type: text/x-rst
 Created: 27-Mar-2018
 Python-Version: 3.8
-Post-History: 31-Mar-2018
+Post-History: 31-Mar-2018, 12-Apr-2018
 
 
 Abstract
@@ -66,7 +66,7 @@ This is the new class hierarchy for functions and methods::
                 /         |   defined_function
                /          |        \
     builtin_function (*)  |         \
-                          |    python_function
+                          |       function
                           |
                     bound_method (*)
 
@@ -224,13 +224,13 @@ And ``__defaults__`` is not required to be used for calling the function.
 Apart from adding these extra attributes,
 ``defined_function`` behaves exactly the same as ``base_function``.
 
-python_function
+function
----------------
+--------
 
 This is the class meant for functions implemented in Python,
 formerly known as ``function``.
 Unlike the other function types,
-instances of ``python_function`` can be created from Python code.
+instances of ``function`` can be created from Python code.
 This is not changed, so we do not describe the details in this PEP.
 
 The layout of the C structure is almost the same as ``defined_function``::
@@ -255,11 +255,11 @@ The layout of the C structure is almost the same as ``defined_function``::
 The only difference is an ``_ml`` field
 which reserves space to be used by ``base.m_ml``.
 
-When constructing an instance of ``python_function`` from ``code`` and ``globals``,
+When constructing an instance of ``function`` from ``code`` and ``globals``,
 an instance is created with ``base.m_ml = &_ml``,
 ``base.m_self = NULL`` and with the ``METH_PYTHON`` flag set.
 
-To make subclassing easier, we also add a copying constructor:
+To make subclassing easier, we also add a copy constructor:
 if ``f`` is an instance of ``defined_function`` with the ``METH_PYTHON``
 flag set, then ``types.FunctionType(f)`` copies ``f``.
 
@@ -445,12 +445,18 @@ because that would not be safe (heap types can be changed dynamically).
 C API functions
 ---------------
 
-We add and change some Python/C API functions:
+We list some relevant Python/C API macros and functions.
+Some of these are existing (possibly changed) functions, some are new:
+
+- ``int PyBaseFunction_CheckFast(PyObject *op)``: return true if ``op``
+  is an instance of a class with the ``Py_TPFLAGS_BASEFUNCTION`` set.
+  This is the function that you need to use to determine
+  whether you can safely access the ``base_function`` internals.
 
 - ``int PyBaseFunction_Check(PyObject *op)``: return true if ``op``
-  is an instance of a type with the ``Py_TPFLAGS_BASEFUNCTION`` set.
+  is an instance of ``base_function``.
 
-- ``PyObject* PyBaseFunction_New(PyTypeObject *cls, PyCFunctionDef *ml, PyObject *self, PyObject *module, PyTypeObject *objclass)``:
+- ``PyObject *PyBaseFunction_New(PyTypeObject *cls, PyCFunctionDef *ml, PyObject *self, PyObject *module, PyTypeObject *objclass)``:
   create a new instance of ``cls`` (which must be a subclass of ``base_function``)
   from the given data.
 
@@ -462,25 +468,32 @@ We add and change some Python/C API functions:
   As special case, if ``self`` is ``NULL``,
   then set ``self = Py_None`` instead (for backwards compatibility).
 
-- ``int PyFunction_Check(PyObject *op)``: return true if ``op``
-  is an instance of ``defined_function``.
-
-- ``PyPythonFunction_New(PyTypeObject *cls, PyObject *code, PyObject *globals, PyObject *name, PyObject *qualname)``:
-  create a new instance of ``cls`` (which must be a sublass of ``python_function``)
-  from the given data.
-
-- ``PyObject* PyFunction_New(PyObject *code, PyObject *globals)``:
-  create a new instance of ``python_function``.
-
-- ``PyObject* PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname)``:
-  create a new instance of ``python_function``.
-
 - For many existing ``PyCFunction_...`` and ``PyMethod_`` functions,
   we define a new function ``PyBaseFunction_...``
   acting on ``base_function`` instances.
   For backwards compatibility,
   the old functions are kept as aliases of the new functions.
 
+- ``int PyFunction_Check(PyObject *op)``: return true if ``op``
+  is an instance of ``defined_function``.
+
+- ``PyObject *PyFunction_NewPython(PyTypeObject *cls, PyObject *code, PyObject *globals, PyObject *name, PyObject *qualname)``:
+  create a new instance of ``cls`` (which must be a sublass of ``function``)
+  from the given data.
+
+- ``PyObject *PyFunction_New(PyObject *code, PyObject *globals)``:
+  create a new instance of ``function``.
+
+- ``PyObject *PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname)``:
+  create a new instance of ``function``.
+
+- ``PyObject *PyFunction_Copy(PyTypeObject *cls, PyObject *func)``:
+  create a new instance of ``cls`` (which must be a sublass of ``function``)
+  by copying a given ``defined_function``.
+
+- All other existing ``PyFunction_...`` functions now act on ``defined_function``
+  instances (instead of ``function``).
+
 Changes to the types module
 ---------------------------
 
@@ -489,7 +502,7 @@ Two types are added: ``types.BaseFunctionType`` corresponding to
 ``defined_function``.
 
 Apart from that, no changes to the ``types`` module are made.
-In particular, ``types.FunctionType`` refers to ``python_function``.
+In particular, ``types.FunctionType`` refers to ``function``.
 However, the actual types will change:
 for example, ``types.BuiltinFunctionType`` will no longer be the same
 as ``types.BuiltinMethodType``.
@@ -539,7 +552,7 @@ Non-CPython implementations
 ===========================
 
 For other implementations of Python apart from CPython,
-only the classes ``base_function``, ``bound_method`` and ``python_function`` are required.
+only the classes ``base_function``, ``bound_method`` and ``function`` are required.
 The latter two are the only classes which can be instantiated directly
 from the Python interpreter.
 We require ``base_function`` for consistency but we put no requirements on it:
@@ -583,28 +596,21 @@ And even if ``__text_signature__`` would allow arbitrary signatures somehow,
 that is only one piece of introspection:
 it does not help with ``inspect.getsourcefile`` for example.
 
-defined_function versus python_function
+defined_function versus function
----------------------------------------
+--------------------------------
-
-The names ``defined_function`` and ``python_function``
-were chosen to be different from ``function``
-because none of the two classes ``defined_function``/``python_function``
-is an obvious candidate to receive the ``function`` name.
-It also allows to use the word "function" informally without referring
-to a specific class.
 
 In many places, a decision needs to be made whether the old ``function`` class
-should be replaced by ``defined_function`` or ``python_function``.
+should be replaced by ``defined_function`` or the new ``function`` class.
 This is done by thinking of the most likely use case:
 
-1. ``types.FunctionType`` refers to ``python_function`` because that
+1. ``types.FunctionType`` refers to ``function`` because that
    type might be used to construct instances using ``types.FunctionType(...)``.
 
 2. ``inspect.isfunction()`` refers to ``defined_function``
    because this is the class where introspection is supported.
 
 3. The C API functions ``PyFunction_New...``
-   refer to ``python_function`` simply because one cannot create instances
+   refer to ``function`` simply because one cannot create instances
    of ``defined_function``.
 
 4. The C API functions ``PyFunction_Check`` and ``PyFunction_Get/Set...``
@@ -678,7 +684,7 @@ to enable fast type checks for ``PyBuiltinFunction_Check`` and ``PyMethod_Check(
 
 We allow subclassing of the other classes because there is no reason to disallow it.
 For Python modules, the only relevant class to subclass is
-``python_function`` because the others cannot be instantiated anyway.
+``function`` because the others cannot be instantiated anyway.
 
 Replacing tp_call: METH_ARG0_FUNCTION
 -------------------------------------
@@ -726,6 +732,11 @@ For Python functions, essentially nothing changes.
 The attributes that existed before still exist and Python functions
 can be initialized, called and turned into methods as before.
 
+The name ``function`` is kept for backwards compatibility.
+While it might make sense to change the name to something more
+specific like ``python_function``,
+that would require a lot of annoying changes in documentation and testsuites.
+
 Built-in functions of a module
 ------------------------------
 
@@ -744,22 +755,28 @@ was specifically designed to be backwards compatible.
 All attributes which existed before (like ``__objclass__`` and ``__self__``)
 still exist.
 
-New classes
+New and changed classes
------------
+-----------------------
 
 Tools which take various kinds of functions as input will need to deal
 with the new function hieararchy and the possibility of custom
 function classes.
-If those tools use ``inspect`` properly, there should be few
+The proposed changes to ``types`` and ``inspect``
-backwards compatibility problems.
+are meant to minimize changes in behaviour.
+However, it is unavoidable that some things change
+and this can cause code to break.
+In the Python standard library,
+changes are needed in the ``doctest`` module because of this.
 
 New attributes
 --------------
 
-Some objects get new attributes.
+Some objects get new special double-underscore attributes.
 For example, ``__objclass__`` now appears on bound methods too
 and all methods get a ``__func__`` attribute.
-We expect that this will not cause problems.
+The fact that ``__self__`` is now a special read-only attribute
+for Python functions caused trouble in [#bpo33265]_.
+Generally, we expect that not much will break though.
 
 method_descriptor and PyDescr_NewMethod
 ---------------------------------------
@@ -772,9 +789,24 @@ They are kept for backwards compatibility though.
 Reference Implementation
 ========================
 
-The implementation in CPython is being developed at
+Most of this PEP has been implemented for CPython at
 https://github.com/jdemeyer/cpython/tree/pep575
 
+There are four steps, corresponding to the commits on that branch.
+After each step, CPython is in a mostly working state.
+
+1. Add the ``base_function`` class and make it a subclass for ``builtin_function``.
+   This is by far the biggest step as the complete ``__call__`` protocol
+   is implemented in this step.
+
+2. Rename ``method`` to ``bound_method`` and make it a subclass of ``base_function``.
+   Change unbound methods of extension types to be instances of ``builtin_function``
+   such that bound methods of extension types are also instances of ``bound_method``.
+
+3. Implement ``defined_function`` and ``function``.
+
+4. Changes to other parts of Python, such as the standard library and testsuite.
+
 
 Appendix: current situation
 ===========================
@@ -902,6 +934,9 @@ References
 .. [#bpo33261] Python bug 33261, inspect.isgeneratorfunction fails on hand-created methods
    (https://bugs.python.org/issue33261 and https://github.com/python/cpython/pull/6448)
 
+.. [#bpo33265] Python bug 33265, contextlib.ExitStack abuses __self__
+   (https://bugs.python.org/issue33265 and https://github.com/python/cpython/pull/6456)
+
 .. [#ABI] PEP 384, Defining a Stable ABI, Löwis (https://www.python.org/dev/peps/pep-0384)
 
 .. [#clinic] PEP 436, The Argument Clinic DSL, Hastings (https://www.python.org/dev/peps/pep-0436)