PEP 670: more examples (#2163)

* Elaborate the Abstract
* Add "Misnesting" example
* The return value has already been removed in the public C API macros
* Clarify that converting Py_TYPE() macro was an incompatible change

pep-0670.rst

@@ -12,7 +12,13 @@ Python-Version: 3.11
 Abstract
 ========
 
-Convert macros to static inline functions or regular functions.
+Convert macros to static inline functions or regular functions to avoid
+macro pitfalls.
+
+Convert macros and static inline functions to regular functions to make
+them usable by Python extensions which cannot use macros or static
+inline functions, like extensions written in a programming languages
+other than C or C++.
 
 Remove the return value of macros having a return value, whereas they
 should not, to aid detecting bugs in C extensions when the C API is
@@ -21,6 +27,9 @@ misused.
 Some function arguments are still cast to ``PyObject*`` to prevent
 emitting new compiler warnings.
 
+Macros which can be used as l-value in an assignment are not converted
+to functions to avoid introducing incompatible changes.
+
 
 Rationale
 =========
@@ -163,9 +172,11 @@ The following macros should not be converted:
 * Compatibility layer for different C compilers, C language extensions,
   or recent C features.
   Example: ``#define Py_ALWAYS_INLINE __attribute__((always_inline))``.
-* Macros that need the stringification or concatenation feature of the C preprocessor.
+* Macros that need C preprocessor features, like stringification and
+  concatenation. Example: ``Py_STRINGIFY()``.
 * Macros which can be used as l-value in an assignment. This change is
-  an incompatible change and it is out of the scope of this PEP.
+  an incompatible change and is out of the scope of this PEP.
+  Example: ``PyBytes_AS_STRING()``.
 
 
 Convert static inline functions to regular functions
@@ -214,11 +225,35 @@ Remove the return value
 -----------------------
 
 When a macro is implemented as an expression, it has an implicit return
-value.  This macro pitfall can be misused in third party C extensions.  See
-`bpo-30459 <https://bugs.python.org/issue30459>`_ regarding the misuse of the
-``PyList_SET_ITEM()`` and ``PyCell_SET()`` macros.  Such pitfalls are hard to
-catch while reviewing macro code.  Removing the return value aids detecting
-bugs in C extensions when the C API is misused.
+value.  This return value can be misused in third party C extensions.
+See `bpo-30459 <https://bugs.python.org/issue30459>`__ regarding the
+misuse of the ``PyList_SET_ITEM()`` and ``PyCell_SET()`` macros.
+
+Such issue is hard to catch while reviewing macro code. Removing the
+return value aids detecting bugs in C extensions when the C API is
+misused.
+
+The issue has already been fixed in public C API macros by the
+`bpo-30459 <https://bugs.python.org/issue30459>`__ in Python 3.10: add a
+``(void)`` cast to the affected macros. Example of the
+``PyTuple_SET_ITEM()`` macro::
+
+    #define PyTuple_SET_ITEM(op, i, v) ((void)(_PyTuple_CAST(op)->ob_item[i] = v))
+
+Example of macros currently using a ``(void)`` cast to have no return
+value:
+
+* ``PyCell_SET()``
+* ``PyList_SET_ITEM()``
+* ``PyTuple_SET_ITEM()``
+* ``Py_BUILD_ASSERT()``
+* ``_PyGCHead_SET_FINALIZED()``
+* ``_PyGCHead_SET_NEXT()``
+* ``_PyObject_ASSERT_FROM()``
+* ``_Py_atomic_signal_fence()``
+* ``_Py_atomic_store_64bit()``
+* ``asdl_seq_SET()``
+* ``asdl_seq_SET_UNTYPED()``
 
 
 Backwards Compatibility
@@ -227,6 +262,9 @@ Backwards Compatibility
 Removing the return value of macros is an incompatible API change made
 on purpose: see the `Remove the return value`_ section.
 
+Some function arguments are still cast to ``PyObject*`` to prevent
+emitting new compiler warnings.
+
 Macros which can be used as l-value in an assignment are not modified by
 this PEP to avoid incompatible changes.
 
@@ -238,9 +276,8 @@ Keep macros, but fix some macro issues
 --------------------------------------
 
 Converting macros to functions is not needed to `remove the return
-value`_: casting a macro return value to ``void`` also fix the issue.
-For example, the ``PyList_SET_ITEM()`` macro was already fixed like
-that.
+value`_: adding a ``(void)`` cast is enough. For example, the
+``PyList_SET_ITEM()`` macro was already fixed like that.
 
 Macros are always "inlined" with any C compiler.
 
@@ -255,8 +292,11 @@ to miss a macro pitfall when writing and reviewing macro code. Moreover, macros
 are harder to read and maintain than functions.
 
 
-Examples of duplication of side effects
-=======================================
+Examples of Macro Pitfalls
+==========================
+
+Duplication of side effects
+---------------------------
 
 Macros::
 
@@ -269,6 +309,49 @@ Macros::
 If the *op* or the *X* argument has a side effect, the side effect is
 duplicated: it executed twice by ``PySet_Check()`` and ``Py_IS_NAN()``.
 
+For example, the ``pos++`` argument in the
+``PyUnicode_WRITE(kind, data, pos++, ch)`` code has a side effect.
+This code is safe because the ``PyUnicode_WRITE()`` macro only uses its
+3rd argument once and so does not duplicate ``pos++`` side effect.
+
+Misnesting
+----------
+
+Example of the `bpo-43181: Python macros don't shield arguments
+<https://bugs.python.org/issue43181>`_. The ``PyObject_TypeCheck()``
+macro before it has been fixed::
+
+    #define PyObject_TypeCheck(ob, tp) \
+        (Py_IS_TYPE(ob, tp) || PyType_IsSubtype(Py_TYPE(ob), (tp)))
+
+C++ usage example::
+
+    PyObject_TypeCheck(ob, U(f<a,b>(c)))
+
+The preprocessor first expands it::
+
+    (Py_IS_TYPE(ob, f<a,b>(c)) || ...)
+
+C++ ``"<"`` and ``">"`` characters are not treated as brackets by the
+preprocessor, so the ``Py_IS_TYPE()`` macro is invoked with 3 arguments:
+
+* ``ob``
+* ``f<a``
+* ``b>(c)``
+
+The compilation fails with an error on ``Py_IS_TYPE()`` which only takes
+2 arguments.
+
+The bug is that the *op* and *tp* arguments of ``PyObject_TypeCheck()``
+must be put between parentheses: replace ``Py_IS_TYPE(ob, tp)`` with
+``Py_IS_TYPE((ob), (tp))``. In regular C code, these parentheses are
+redundant, can be seen as a bug, and so are often forgotten when writing
+macros.
+
+To avoid Macro Pitfalls, the ``PyObject_TypeCheck()`` macro has been
+converted to a static inline function:
+`commit <https://github.com/python/cpython/commit/4bb2a1ebc569eee6f1b46ecef1965a26ae8cb76d>`__.
+
 
 Examples of hard to read macros
 ===============================
@@ -303,7 +386,7 @@ Python 3.8 function (simplified code)::
   as a single long line.
 * Inside the function, the *op* argument has the well defined type
   ``PyObject*`` and so doesn't need casts like ``(PyObject *)(op)``.
-* Arguments don't need to be put inside parenthesis: use ``typeobj``,
+* Arguments don't need to be put inside parentheses: use ``typeobj``,
   rather than ``(typeobj)``.
 
 _Py_NewReference()
@@ -375,6 +458,12 @@ Possible implementation as a static inlined function::
 Macros converted to functions since Python 3.8
 ==============================================
 
+List of macros already converted to functions between Python 3.8 and
+Python 3.11 showing that these conversions didn't not impact the Python
+performance and didn't break the backward compatibility, even if some
+converted macros are very commonly used by C extensions like
+``Py_INCREF()``.
+
 Macros converted to static inline functions
 -------------------------------------------
 
@@ -389,15 +478,6 @@ Python 3.8:
 * ``_PyObject_GC_UNTRACK()``
 * ``_Py_Dealloc()``
 
-Python 3.10:
-
-* ``Py_REFCNT()``
-
-Python 3.11:
-
-* ``Py_TYPE()``
-* ``Py_SIZE()``
-
 Macros converted to regular functions
 -------------------------------------
 
@@ -418,6 +498,7 @@ private static inline functions have been added to the internal C API:
 * ``_PyType_HasFeature()``
 * ``_PyType_IS_GC()``
 
+
 Static inline functions converted to regular functions
 -------------------------------------------------------
 
@@ -434,6 +515,22 @@ private static inline function has been added to the internal C API:
 * ``_PyVectorcall_FunctionInline()``
 
 
+Incompatible changes
+--------------------
+
+While other converted macros didn't break the backward compatibility,
+there are is an exception.
+
+The 3 macros ``Py_REFCNT()``, ``Py_TYPE()`` and ``Py_SIZE()`` have been
+converted to static inline functions in Python 3.10 and 3.11 to disallow
+using them as l-value in assignment. It is an incompatible change made
+on purpose: see `bpo-39573 <https://bugs.python.org/issue39573>`_ for
+the rationale.
+
+This PEP does not convert macros which can be used as l-value to avoid
+introducing incompatible changes.
+
+
 Benchmark comparing macros and static inline functions
 ======================================================