PEP 445

pep-0445.txt

@@ -46,54 +46,85 @@ API changes
   - ``void* PyMem_RawMalloc(size_t size)``
   - ``void* PyMem_RawRealloc(void *ptr, size_t new_size)``
   - ``void PyMem_RawFree(void *ptr)``
+  - the behaviour of requesting zero bytes is not defined: return *NULL* or a
+    distinct non-*NULL* pointer depending on the platform.
 
-* Add a new ``PyMemAllocators`` structure::
+* Add a new ``PyMemBlockAllocator`` structure::
 
     typedef struct {
         /* user context passed as the first argument to the 3 functions */
         void *ctx;
 
-        /* allocate memory */
+        /* allocate a memory block */
         void* (*malloc) (void *ctx, size_t size);
 
-        /* allocate memory or resize a memory buffer */
+        /* allocate or resize a memory block */
         void* (*realloc) (void *ctx, void *ptr, size_t new_size);
 
-        /* release memory */
+        /* release a memory block */
         void (*free) (void *ctx, void *ptr);
-    } PyMemAllocators;
+    } PyMemBlockAllocator;
 
 * Add new functions to get and set memory block allocators:
 
-  - ``void PyMem_GetRawAllocators(PyMemAllocators *allocators)``
+  - Get/Set internal functions of ``PyMem_RawMalloc()``,
-  - ``void PyMem_SetRawAllocators(PyMemAllocators *allocators)``
+    ``PyMem_RawRealloc()`` and ``PyMem_RawFree()``:
-  - ``void PyMem_GetAllocators(PyMemAllocators *allocators)``
-  - ``void PyMem_SetAllocators(PyMemAllocators *allocators)``
-  - ``void PyObject_GetAllocators(PyMemAllocators *allocators)``
-  - ``void PyObject_SetAllocators(PyMemAllocators *allocators)``
 
-* Add new functions to get and set memory mapping allocators:
+    * ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)``
+    * ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)``
 
-  - ``void _PyObject_GetArenaAllocators(void **ctx_p, void* (**malloc_p) (void *ctx, size_t size), void (**free_p) (void *ctx, void *ptr, size_t size))``
+  - Get/Set internal functions of ``PyMem_Malloc()``,
-  - ``void _PyObject_SetArenaAllocators(void *ctx, void* (*malloc) (void *ctx, size_t size), void (*free) (void *ctx, void *ptr, size_t size))``
+    ``PyMem_Realloc()`` and ``PyMem_Free()``:
+
+    * ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)``
+    * ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)``
+    * ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return *NULL*:
+      it would be treated as an error.
+
+  - Get/Set internal functions of ``PyObject_Malloc()``,,
+    ``PyObject_Realloc()`` and ``PyObject_Free()``:
+
+    * ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)``
+    * ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)``
+
+* Add a new ``PyMemMappingAllocator`` structure::
+
+    typedef struct {
+        /* user context passed as the first argument to the 2 functions */
+        void *ctx;
+
+        /* allocate a memory mapping */
+        void* (*malloc) (void *ctx, size_t size);
+
+        /* release a memory mapping */
+        void (*free) (void *ctx, void *ptr, size_t size);
+    } PyMemMappingAllocator;
+
+* Add a new function to get and set memory mapping allocator:
+
+  - ``void PyMem_GetMappingAllocator(PyMemMappingAllocator *allocator)``
+  - ``void PyMem_SetMappingAllocator(PyMemMappingAllocator *allocator)``
+  - Currently, this allocator is only used internally by *pymalloc* to allocate
+    arenas.
 
 * Add a new function to setup the builtin Python debug hooks when memory
   allocators are replaced:
 
   - ``void PyMem_SetupDebugHooks(void)``
 
-.. note::
-
 The builtin Python debug hooks were introduced in Python 2.3 and implement the
 following checks:
 
-   * Newly allocated memory is filled with the byte 0xCB, freed memory is filled
+* Newly allocated memory is filled with the byte ``0xCB``, freed memory is
-     with the byte 0xDB.
+  filled with the byte ``0xDB``.
 * Detect API violations, ex: ``PyObject_Free()`` called on a memory block
   allocated by ``PyMem_Malloc()``
 * Detect write before the start of the buffer (buffer underflow)
 * Detect write after the end of the buffer (buffer overflow)
 
+The *pymalloc* allocator is used by default for:
+``PyObject_Malloc()``, ``PyObject_Realloc()`` and ``PyObject_Free()``.
+
 
 Make usage of these new APIs
 ----------------------------
@@ -117,10 +148,10 @@ Make usage of these new APIs
 Examples
 ========
 
-Use case 1: Replace Memory Allocators, keep pymalloc
+Use case 1: Replace Memory Allocator, keep pymalloc
 ----------------------------------------------------
 
-Setup your custom memory allocators, keeping pymalloc. Dummy example wasting 2
+Setup your custom memory allocator, keeping pymalloc. Dummy example wasting 2
 bytes per allocation, and 10 bytes per arena::
 
     #include <stdlib.h>
@@ -156,30 +187,30 @@ bytes per allocation, and 10 bytes per arena::
         free(ptr);
     }
 
-    void setup_custom_allocators(void)
+    void setup_custom_allocator(void)
     {
-        PyMemAllocators alloc;
+        PyMemBlockAllocator alloc;
 
         alloc.ctx = &alloc_padding;
         alloc.malloc = my_malloc;
         alloc.realloc = my_realloc;
         alloc.free = my_free;
 
-        PyMem_SetRawAllocators(&alloc);
+        PyMem_SetRawAllocator(&alloc);
-        PyMem_SetAllocators(&alloc);
+        PyMem_SetAllocator(&alloc);
 
-        _PyObject_SetArenaAllocators(&arena_padding,
+        _PyObject_SetArenaAllocator(&arena_padding,
                                      my_alloc_arena, my_free_arena);
 
         PyMem_SetupDebugHooks();
     }
 
 .. warning::
-   Remove the call ``PyMem_SetRawAllocators(&alloc)`` if the new allocators
+   Remove the call ``PyMem_SetRawAllocator(&alloc)`` if the new allocator
    are not thread-safe.
 
 
-Use case 2: Replace Memory Allocators, override pymalloc
+Use case 2: Replace Memory Allocator, override pymalloc
 --------------------------------------------------------
 
 If your allocator is optimized for allocation of small objects (less than 512
@@ -209,23 +240,23 @@ Dummy Example wasting 2 bytes per allocation::
         free(ptr);
     }
 
-    void setup_custom_allocators(void)
+    void setup_custom_allocator(void)
     {
-        PyMemAllocators alloc;
+        PyMemBlockAllocator alloc;
         alloc.ctx = &padding;
         alloc.malloc = my_malloc;
         alloc.realloc = my_realloc;
         alloc.free = my_free;
 
-        PyMem_SetRawAllocators(&alloc);
+        PyMem_SetRawAllocator(&alloc);
-        PyMem_SetAllocators(&alloc);
+        PyMem_SetAllocator(&alloc);
-        PyObject_SetAllocators(&alloc);
+        PyObject_SetAllocator(&alloc);
 
         PyMem_SetupDebugHooks();
     }
 
 .. warning::
-   Remove the call ``PyMem_SetRawAllocators(&alloc)`` if the new allocators
+   Remove the call ``PyMem_SetRawAllocator(&alloc)`` if the new allocator
    are not thread-safe.
 
 
@@ -236,15 +267,15 @@ Use case 3: Setup Allocator Hooks
 Example to setup hooks on all memory allocators::
 
     struct {
-        PyMemAllocators pymem;
+        PyMemBlockAllocator pymem;
-        PyMemAllocators pymem_raw;
+        PyMemBlockAllocator pymem_raw;
-        PyMemAllocators pyobj;
+        PyMemBlockAllocator pyobj;
         /* ... */
     } hook;
 
     static void* hook_malloc(void *ctx, size_t size)
     {
-        PyMemAllocators *alloc = (PyMemAllocators *)ctx;
+        PyMemBlockAllocator *alloc = (PyMemBlockAllocator *)ctx;
         /* ... */
         ptr = alloc->malloc(alloc->ctx, size);
         /* ... */
@@ -253,7 +284,7 @@ Example to setup hooks on all memory allocators::
 
     static void* hook_realloc(void *ctx, void *ptr, size_t new_size)
     {
-        PyMemAllocators *alloc = (PyMemAllocators *)ctx;
+        PyMemBlockAllocator *alloc = (PyMemBlockAllocator *)ctx;
         void *ptr2;
         /* ... */
         ptr2 = alloc->realloc(alloc->ctx, ptr, new_size);
@@ -263,7 +294,7 @@ Example to setup hooks on all memory allocators::
 
     static void hook_free(void *ctx, void *ptr)
     {
-        PyMemAllocators *alloc = (PyMemAllocators *)ctx;
+        PyMemBlockAllocator *alloc = (PyMemBlockAllocator *)ctx;
         /* ... */
         alloc->free(alloc->ctx, ptr);
         /* ... */
@@ -271,7 +302,7 @@ Example to setup hooks on all memory allocators::
 
     void setup_hooks(void)
     {
-        PyMemAllocators alloc;
+        PyMemBlockAllocator alloc;
         static int installed = 0;
 
         if (installed)
@@ -282,21 +313,21 @@ Example to setup hooks on all memory allocators::
         alloc.realloc = hook_realloc;
         alloc.free = hook_free;
 
-        PyMem_GetRawAllocators(&hook.pymem_raw);
+        PyMem_GetRawAllocator(&hook.pymem_raw);
         alloc.ctx = &hook.pymem_raw;
-        PyMem_SetRawAllocators(&alloc);
+        PyMem_SetRawAllocator(&alloc);
 
-        PyMem_GetAllocators(&hook.pymem);
+        PyMem_GetAllocator(&hook.pymem);
         alloc.ctx = &hook.pymem;
-        PyMem_SetAllocators(&alloc);
+        PyMem_SetAllocator(&alloc);
 
-        PyObject_GetAllocators(&hook.pyobj);
+        PyObject_GetAllocator(&hook.pyobj);
         alloc.ctx = &hook.pyobj;
-        PyObject_SetAllocators(&alloc);
+        PyObject_SetAllocator(&alloc);
     }
 
 .. warning::
-   Remove the call ``PyMem_SetRawAllocators(&alloc)`` if hooks are not
+   Remove the call ``PyMem_SetRawAllocator(&alloc)`` if hooks are not
    thread-safe.
 
 .. note::
@@ -326,13 +357,19 @@ Only have one generic get/set function
 
 Replace the 6 functions:
 
-* ``PyMem_GetRawAllocators()``, ``PyMem_GetAllocators()``, ``PyObject_GetAllocators()``
+* ``PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)``
-* ``PyMem_SetRawAllocators(allocators)``, ``PyMem_SetAllocators(allocators)``, ``PyObject_SetAllocators(allocators)``
+* ``PyMem_GetAllocator(PyMemBlockAllocator *allocator)``
+* ``PyObject_GetAllocator(PyMemBlockAllocator *allocator)``
+* ``PyMem_SetRawAllocator(allocator)``
+* ``PyMem_SetAllocator(PyMemBlockAllocator *allocator)``
+* ``PyObject_SetAllocator(PyMemBlockAllocator *allocator)``
 
 with 2 functions with an additional *domain* argument:
 
-* ``Py_GetAllocators(domain)``
+* ``int Py_GetAllocator(int domain, PyMemBlockAllocator *allocator)``
-* ``Py_SetAllocators(domain, allocators)``
+* ``int Py_SetAllocator(int domain, PyMemBlockAllocator *allocator)``
+
+These functions return 0 on success, or -1 if the domain is unknown.
 
 where domain is one of these values:
 
@@ -341,9 +378,19 @@ where domain is one of these values:
 * ``PYALLOC_PYOBJECT``
 
 
-``_PyObject_GetArenaAllocators()`` and ``_PyObject_SetArenaAllocators()`` are
+PyMem_Malloc() reuses PyMem_RawMalloc() by default
-not merged and kept private because their prototypes are different and they are
+--------------------------------------------------
-specific to pymalloc.
+
+``PyMem_Malloc()`` should call ``PyMem_RawMalloc()`` by default. So calling
+``PyMem_SetRawAllocator()`` would also also patch ``PyMem_Malloc()``
+indirectly.
+
+Such change is less optimal, it adds another level of indirection.
+
+In the proposed implementation of this PEP (issue #3329), ``PyMem_RawMalloc()``
+calls directly ``malloc()``, whereas ``PyMem_Malloc()`` returns ``NULL`` if
+size is larger than ``PY_SSIZE_T_MAX``, and the default allocator of
+``PyMem_Malloc()`` calls ``malloc(1)`` if the size is zero.
 
 
 Add a new PYDEBUGMALLOC environment variable
@@ -353,8 +400,8 @@ To be able to use the Python builtin debug hooks even when a custom memory
 allocator replaces the default Python allocator, an environment variable
 ``PYDEBUGMALLOC`` can be added to setup these debug function hooks, instead of
 adding the new function ``PyMem_SetupDebugHooks()``. If the environment
-variable is present, ``PyMem_SetRawAllocators()``, ``PyMem_SetAllocators()``
+variable is present, ``PyMem_SetRawAllocator()``, ``PyMem_SetAllocator()``
-and ``PyObject_SetAllocators()`` will reinstall automatically the hook on top
+and ``PyObject_SetAllocator()`` will reinstall automatically the hook on top
 of the new allocator.
 
 An new environment variable would make the Python initialization even more
@@ -386,21 +433,6 @@ objects of differenet types would have the same allocation location. Such
 changes add too much complexity for a little gain.
 
 
-No context argument
--------------------
-
-Simplify the signature of allocator functions, remove the context argument:
-
-* ``void* malloc(size_t size)``
-* ``void* realloc(void *ptr, size_t new_size)``
-* ``void free(void *ptr)``
-
-It is likely for an allocator hook to be reused for ``PyMem_SetAllocators()``
-and ``PyObject_SetAllocators()``, but the hook must call a different function
-depending on the allocator. The context is a convenient way to reuse the same
-allocator or hook for different APIs.
-
-
 PyMem_Malloc() GIL-free
 -----------------------
 
@@ -436,7 +468,6 @@ be seen. Remaining ``malloc()`` may allocate a lot of memory and so would be
 missed in reports.
 
 
-
 Use existing debug tools to analyze the memory
 ----------------------------------------------
 
@@ -458,6 +489,36 @@ information. Being able to setup a hook on allocators called with the GIL held
 allow to read a lot of useful data from Python internals.
 
 
+Add msize()
+-----------
+
+Add another field to ``PyMemBlockAllocator`` and ``PyMemMappingAllocator``::
+
+    size_t msize(void *ptr);
+
+This function returns the size of a memory block or a memory mapping. Return
+(size_t)-1 if the function is not implemented or if the pointer is unknown
+(ex: NULL pointer).
+
+On Windows, this function can be implemented using ``_msize()`` and
+``VirtualQuery()``.
+
+
+No context argument
+-------------------
+
+Simplify the signature of allocator functions, remove the context argument:
+
+* ``void* malloc(size_t size)``
+* ``void* realloc(void *ptr, size_t new_size)``
+* ``void free(void *ptr)``
+
+It is likely for an allocator hook to be reused for ``PyMem_SetAllocator()``
+and ``PyObject_SetAllocator()``, but the hook must call a different function
+depending on the allocator. The context is a convenient way to reuse the same
+allocator or hook for different APIs.
+
+
 External libraries
 ==================
 
@@ -476,7 +537,6 @@ See also the `GNU libc: Memory Allocation Hooks
 <http://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html>`_.
 
 
-
 Memory allocators
 =================
 
@@ -490,18 +550,24 @@ tcmalloc which is part of `gperftools <http://code.google.com/p/gperftools/>`_.
 mappings are usually used for large allocations (ex: larger than 256 KB),
 whereas the heap is used for small allocations.
 
-The heap is handled by ``brk()`` and ``sbrk()`` system calls on Linux, and is
+On UNIX, the heap is handled by ``brk()`` and ``sbrk()`` system calls on Linux,
-contiguous.  Memory mappings are handled by ``mmap()`` on UNIX and
+and it is contiguous.  On Windows, the heap is handled by ``HeapAlloc()`` and
+may be discontiguous. Memory mappings are handled by ``mmap()`` on UNIX and
 ``VirtualAlloc()`` on Windows, they may be discontiguous.
 
 Releasing a memory mapping gives back immediatly the memory to the system. For
 the heap, memory is only given back to the system if it is at the end of the
 heap. Otherwise, the memory will only be given back to the system when all the
-memory located after the released memory are also released. To allocate memory
+memory located after the released memory are also released.
-in the heap, the allocator tries to reuse free space. If there is no contiguous
+
-space big enough, the heap must be increased, even if we have more free space
+To allocate memory in the heap, the allocator tries to reuse free space. If
-than required size.  This issue is called the "memory fragmentation": the
+there is no contiguous space big enough, the heap must be increased, even if we
-memory usage seen by the system may be much higher than real usage.
+have more free space than required size.  This issue is called the "memory
+fragmentation": the memory usage seen by the system may be much higher than
+real usage.
+
+On Windows, ``HeapAlloc()`` creates a new memory mapping with
+``VirtualAlloc()`` if there is not enough free contiguous memory.
 
 CPython has a pymalloc allocator using arenas of 256 KB for allocations smaller
 than 512 bytes. This allocator is optimized for small objects with a short