PEP 445

pep-0445.txt

@@ -12,7 +12,7 @@ Python-Version: 3.4
 Abstract
 ========
 
-Add new APIs to customize memory allocators
+Add new APIs to customize memory allocators.
 
 
 Rationale
@@ -20,15 +20,19 @@ Rationale
 
 Use cases:
 
-* Application embedding Python wanting to use a custom memory allocator
+* Application embedding Python may want to isolate Python memory from the
-  to allocate all Python memory somewhere else or with a different algorithm
+  memory of the application, or may want to different memory allocator
+  optimized for its Python usage
 * Python running on embedded devices with low memory and slow CPU.
   A custom memory allocator may be required to use efficiently the memory
   and/or to be able to use all memory of the device.
-* Debug tool to track memory leaks
+* Debug tool to:
-* Debug tool to detect buffer underflow, buffer overflow and misuse
+
-  of Python allocator APIs
+  - track memory leaks
-* Debug tool to inject bugs, simulate out of memory for example
+  - get the Python filename and line number where an object was allocated
+  - detect buffer underflow, buffer overflow and detect misuse of Python
+    allocator APIs (builtin Python debug hooks)
+  - force allocation to fail to test handling of ``MemoryError`` exception
 
 API:
 
@@ -62,8 +66,8 @@ API changes
   - ``void _PyObject_GetArenaAllocators(void **ctx_p, void* (**malloc_p) (void *ctx, size_t size), void (**free_p) (void *ctx, void *ptr, size_t size))``
   - ``void _PyObject_SetArenaAllocators(void *ctx, void* (*malloc) (void *ctx, size_t size), void (*free) (void *ctx, void *ptr, size_t size))``
 
-* Add a new function to setup debug hooks after memory allocators were
+* Add a new function to setup Python builtin debug hooks when memory
-  replaced:
+  allocators are replaced:
 
   - ``void PyMem_SetupDebugHooks(void)``
 
@@ -71,19 +75,19 @@ API changes
 Use these new APIs
 ------------------
 
-* ``PyMem_Malloc()`` and ``PyMem_Realloc()`` now always call ``malloc()`` and
+* ``PyMem_Malloc()`` and ``PyMem_Realloc()`` always call ``malloc()`` and
   ``realloc()``, instead of calling ``PyObject_Malloc()`` and
   ``PyObject_Realloc()`` in debug mode
 
-* ``PyObject_Malloc()`` now falls back on ``PyMem_Malloc()`` instead of
+* ``PyObject_Malloc()`` falls back on ``PyMem_Malloc()`` instead of
-  ``malloc()`` if size is bigger than ``SMALL_REQUEST_THRESHOLD``, and
+  ``malloc()`` if size is greater or equal than ``SMALL_REQUEST_THRESHOLD``
-  ``PyObject_Realloc()`` falls back on ``PyMem_Realloc()`` instead of
+  (512 bytes), and ``PyObject_Realloc()`` falls back on ``PyMem_Realloc()``
-  ``realloc()``
+  instead of ``realloc()``
 
 * Replace direct calls to ``malloc()`` with ``PyMem_Malloc()``, or
   ``PyMem_RawMalloc()`` if the GIL is not held
 
-* Configure external libraries like zlib or OpenSSL to use
+* Configure external libraries like zlib or OpenSSL to allocate memory using
   ``PyMem_RawMalloc()``
 
 
@@ -93,7 +97,8 @@ Examples
 Use case 1: Replace Memory Allocators, keep pymalloc
 ----------------------------------------------------
 
-Setup your custom memory allocators, keeping pymalloc::
+Setup your custom memory allocators, keeping pymalloc. Dummy example wasting 2
+bytes per allocation, and 10 bytes per arena::
 
     #include <stdlib.h>
 
@@ -156,7 +161,9 @@ Use case 2: Replace Memory Allocators, overriding pymalloc
 
 If your allocator is optimized for allocation of small objects (less than 512
 bytes) with a short liftime, you can replace override pymalloc (replace
-``PyObject_Malloc()``).  Example::
+``PyObject_Malloc()``).
+
+Dummy Example wasting 2 bytes per allocation::
 
     #include <stdlib.h>
 
@@ -203,7 +210,7 @@ bytes) with a short liftime, you can replace override pymalloc (replace
 Use case 3: Setup Allocator Hooks
 ---------------------------------
 
-Setup hooks on memory allocators::
+Example to setup hooks on memory allocators::
 
     struct {
         PyMemAllocators pymem;
@@ -314,13 +321,20 @@ where domain is one of these values:
 * ``PYALLOC_PYOBJECT``
 
 
-Setup Builtin Debug Hooks
+Add a new PYDEBUGMALLOC environment variable
--------------------------
+--------------------------------------------
 
-To be able to use Python debug functions (like ``_PyMem_DebugMalloc()``) even
+To be able to use Python builtin debug hooks even when a custom memory
-when a custom memory allocator is set, an environment variable
+allocator is set, an environment variable ``PYDEBUGMALLOC`` can be added to
-``PYDEBUGMALLOC`` can be added to set these debug function hooks, instead of
+setup these debug function hooks, instead of adding the new function
-the new function ``PyMem_SetupDebugHooks()``.
+``PyMem_SetupDebugHooks()``. If the environment variable is present,
+``PyMem_SetRawAllocators()``, ``PyMem_SetAllocators()`` and
+``PyObject_SetAllocators()`` will reinstall automatically the hook on top of
+the new allocator.
+
+An new environment variable would make the Python initialization even more
+complex. The `PEP 432 <http://www.python.org/dev/peps/pep-0432/>`_ tries to
+simply the CPython startup sequence.
 
 
 Use macros to get customizable allocators
@@ -336,6 +350,12 @@ Pass the C filename and line number
 Use C macros using ``__FILE__`` and ``__LINE__`` to get the C filename
 and line number of a memory allocation.
 
+Passing a filename and a line number to each allocator makes the API more
+complex: pass 3 new arguments instead of just a context argument, to each
+allocator function. GC allocator functions should also be patched,
+``_PyObject_GC_Malloc()`` is used in many C functions for example. Such changes
+add too much complexity, for a little gain.
+
 
 No context argument
 -------------------
@@ -446,7 +466,7 @@ CPython issues related to memory allocation:
   <http://bugs.python.org/issue13483>`_
 * `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which
   isn't thread safe <http://bugs.python.org/issue16742>`_
-* `Issue #18203: Replace calls to malloc() with PyMem_Malloc()
+* `Issue #18203: Replace calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc()
   <http://bugs.python.org/issue18203>`_
 * `Issue #18227: Use Python memory allocators in external libraries like zlib
   or OpenSSL <http://bugs.python.org/issue18227>`_