PEP: 445 Title: Add new APIs to customize memory allocators Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-june-2013 Python-Version: 3.4 Abstract ======== Add new APIs to customize memory allocators Rationale ========= Use cases: * Application embedding Python wanting to use a custom memory allocator to allocate all Python memory somewhere else or with a different algorithm * Python running on embedded devices with low memory and slow CPU. A custom memory allocator may be required to use efficiently the memory and/or to be able to use all memory of the device. * Debug tool to track memory leaks * Debug tool to detect buffer underflow, buffer overflow and misuse of Python allocator APIs * Debug tool to inject bugs, simulate out of memory for example API: * Setup a custom memory allocator for all memory allocated by Python * Hook memory allocator functions to call extra code before and/or after the underlying allocator function Proposal ======== API changes ----------- * Add a new ``PyMemAllocators`` structure * Add new GIL-free memory allocator functions: - ``void* PyMem_RawMalloc(size_t size)`` - ``void* PyMem_RawRealloc(void *ptr, size_t new_size)`` - ``void PyMem_RawFree(void *ptr)`` * Add new functions to get and set memory allocators: - ``void PyMem_GetRawAllocators(PyMemAllocators *allocators)`` - ``void PyMem_SetRawAllocators(PyMemAllocators *allocators)`` - ``void PyMem_GetAllocators(PyMemAllocators *allocators)`` - ``void PyMem_SetAllocators(PyMemAllocators *allocators)`` - ``void PyObject_GetAllocators(PyMemAllocators *allocators)`` - ``void PyObject_SetAllocators(PyMemAllocators *allocators)`` - ``void _PyObject_GetArenaAllocators(void **ctx_p, void* (**malloc_p) (void *ctx, size_t size), void (**free_p) (void *ctx, void *ptr, size_t size))`` - ``void _PyObject_SetArenaAllocators(void *ctx, void* (*malloc) (void *ctx, size_t size), void (*free) (void *ctx, void *ptr, size_t size))`` * Add a new function to setup debug hooks after memory allocators were replaced: - ``void PyMem_SetupDebugHooks(void)`` Use these new APIs ------------------ * ``PyMem_Malloc()`` and ``PyMem_Realloc()`` now always call ``malloc()`` and ``realloc()``, instead of calling ``PyObject_Malloc()`` and ``PyObject_Realloc()`` in debug mode * ``PyObject_Malloc()`` now falls back on ``PyMem_Malloc()`` instead of ``malloc()`` if size is bigger than ``SMALL_REQUEST_THRESHOLD``, and ``PyObject_Realloc()`` falls back on ``PyMem_Realloc()`` instead of ``realloc()`` * Replace direct calls to ``malloc()`` with ``PyMem_Malloc()``, or ``PyMem_RawMalloc()`` if the GIL is not held * Configure external libraries like zlib or OpenSSL to use ``PyMem_RawMalloc()`` Examples ======== Use case 1: Replace Memory Allocators, keep pymalloc ---------------------------------------------------- Setup your custom memory allocators, keeping pymalloc:: #include int alloc_padding = 2; int arena_padding = 10; void* my_malloc(void *ctx, size_t size) { int padding = *(int *)ctx; return malloc(size + padding); } void* my_realloc(void *ctx, void *ptr, size_t new_size) { int padding = *(int *)ctx; return realloc(ptr, new_size + padding); } void my_free(void *ctx, void *ptr) { free(ptr); } void* my_alloc_arena(void *ctx, size_t size) { int padding = *(int *)ctx; return malloc(size + padding); } void my_free_arena(void *ctx, void *ptr, size_t size) { free(ptr); } void setup_custom_allocators(void) { PyMemAllocators alloc; alloc.ctx = &alloc_padding; alloc.malloc = my_malloc; alloc.realloc = my_realloc; alloc.free = my_free; PyMem_SetRawAllocators(&alloc); PyMem_SetAllocators(&alloc); _PyObject_SetArenaAllocators(&arena_padding, my_alloc_arena, my_free_arena); PyMem_SetupDebugHooks(); } .. warning:: Remove the call ``PyMem_SetRawAllocators(&alloc)`` if the new allocators are not thread-safe. Use case 2: Replace Memory Allocators, overriding pymalloc ---------------------------------------------------------- If your allocator is optimized for allocation of small objects (less than 512 bytes) with a short liftime, you can replace override pymalloc (replace ``PyObject_Malloc()``). Example:: #include int padding = 2; void* my_malloc(void *ctx, size_t size) { int padding = *(int *)ctx; return malloc(size + padding); } void* my_realloc(void *ctx, void *ptr, size_t new_size) { int padding = *(int *)ctx; return realloc(ptr, new_size + padding); } void my_free(void *ctx, void *ptr) { free(ptr); } void setup_custom_allocators(void) { PyMemAllocators alloc; alloc.ctx = &padding; alloc.malloc = my_malloc; alloc.realloc = my_realloc; alloc.free = my_free; PyMem_SetRawAllocators(&alloc); PyMem_SetAllocators(&alloc); PyObject_SetAllocators(&alloc); PyMem_SetupDebugHooks(); } .. warning:: Remove the call ``PyMem_SetRawAllocators(&alloc)`` if the new allocators are not thread-safe. Use case 3: Setup Allocator Hooks --------------------------------- Setup hooks on memory allocators:: struct { PyMemAllocators pymem; PyMemAllocators pymem_raw; PyMemAllocators pyobj; /* ... */ } hook; static void* hook_malloc(void *ctx, size_t size) { PyMemAllocators *alloc = (PyMemAllocators *)ctx; /* ... */ ptr = alloc->malloc(alloc->ctx, size); /* ... */ return ptr; } static void* hook_realloc(void *ctx, void *ptr, size_t new_size) { PyMemAllocators *alloc = (PyMemAllocators *)ctx; void *ptr2; /* ... */ ptr2 = alloc->realloc(alloc->ctx, ptr, new_size); /* ... */ return ptr2; } static void hook_free(void *ctx, void *ptr) { PyMemAllocators *alloc = (PyMemAllocators *)ctx; /* ... */ alloc->free(alloc->ctx, ptr); /* ... */ } void setup_hooks(void) { PyMemAllocators alloc; static int registered = 0; if (registered) return; registered = 1; alloc.malloc = hook_malloc; alloc.realloc = hook_realloc; alloc.free = hook_free; PyMem_GetRawAllocators(&hook.pymem_raw); alloc.ctx = &hook.pymem_raw; PyMem_SetRawAllocators(&alloc); PyMem_GetAllocators(&hook.pymem); alloc.ctx = &hook.pymem; PyMem_SetAllocators(&alloc); PyObject_GetAllocators(&hook.pyobj); alloc.ctx = &hook.pyobj; PyObject_SetAllocators(&alloc); } .. warning:: Remove the call ``PyMem_SetRawAllocators(&alloc)`` if hooks are not thread-safe. .. note:: ``PyMem_SetupDebugHooks()`` does not need to be called: Python debug hooks are installed automatically at startup. Performances ============ The `Python benchmarks suite `_ (-b 2n3): some tests are 1.04x faster, some tests are 1.04 slower, significant is between 115 and -191. I don't understand these output, but I guess that the overhead cannot be seen with such test. pybench: "+0.1%" (diff between -4.9% and +5.6%). The full output is attached to the issue #3329. Alternatives ============ Only one get and one set function --------------------------------- Replace the 6 functions: * ``PyMem_GetRawAllocators()`` * ``PyMem_GetAllocators()`` * ``PyObject_GetAllocators()`` * ``PyMem_SetRawAllocators(allocators)`` * ``PyMem_SetAllocators(allocators)`` * ``PyObject_SetAllocators(allocators)`` with 2 functions with an additional *domain* argument: * ``Py_GetAllocators(domain)`` * ``Py_SetAllocators(domain, allocators)`` where domain is one of these values: * ``PYALLOC_PYMEM`` * ``PYALLOC_PYMEM_RAW`` * ``PYALLOC_PYOBJECT`` Setup Builtin Debug Hooks ------------------------- To be able to use Python debug functions (like ``_PyMem_DebugMalloc()``) even when a custom memory allocator is set, an environment variable ``PYDEBUGMALLOC`` can be added to set these debug function hooks, instead of the new function ``PyMem_SetupDebugHooks()``. Use macros to get customizable allocators ----------------------------------------- To have no overhead in the default configuration, customizable allocators would be an optional feature enabled by a configuration option or by macros. Pass the C filename and line number ----------------------------------- Use C macros using ``__FILE__`` and ``__LINE__`` to get the C filename and line number of a memory allocation. No context argument ------------------- Simplify the signature of allocator functions, remove the context argument: * ``void* malloc(size_t size)`` * ``void* realloc(void *ptr, size_t new_size)`` * ``void free(void *ptr)`` The context is a convenient way to reuse the same allocator for different APIs (ex: PyMem and PyObject). PyMem_Malloc() GIL-free ----------------------- There is no real reason to require the GIL when calling ``PyMem_Malloc()``. Allowing to call ``PyMem_Malloc()`` without holding the GIL might break applications which setup their own allocator or their allocator hooks. Holding the GIL is very convinient to develop a custom allocator or a hook (no need to care of other threads, no need to handle mutexes, etc.). Don't add PyMem_RawMalloc() --------------------------- Replace ``malloc()`` with ``PyMem_Malloc()``, but if the GIL is not held: keep ``malloc()`` unchanged. The ``PyMem_Malloc()`` is sometimes already misused. For example, the ``main()`` and ``Py_Main()`` functions of Python call ``PyMem_Malloc()`` whereas the GIL do not exist yet. In this case, ``PyMem_Malloc()`` should be replaced with ``malloc()``. If an hook is used to the track memory usage, the ``malloc()`` memory will not be seen. Remaining ``malloc()`` may allocate a lot of memory and so would be missed in reports. CCP API ------- XXX To be done (Kristján Valur Jónsson) XXX External libraries ================== * glib: `g_mem_set_vtable() `_ See also the `GNU libc: Memory Allocation Hooks `_. Memory allocators ================= The C standard library provides the well known ``malloc()`` function. Its implementation depends on the platform and of the C library. The GNU C library uses a modified ptmalloc2, based on "Doug Lea's Malloc" (dlmalloc). FreeBSD uses `jemalloc `_. Google provides tcmalloc which is part of `gperftools `_. ``malloc()`` uses two kinds of memory: heap and memory mappings. Memory mappings are usually used for large allocations (ex: larger than 256 KB), whereas the heap is used for small allocations. The heap is handled by ``brk()`` and ``sbrk()`` system calls on Linux, and is contiguous. Memory mappings are handled by ``mmap()`` on UNIX and ``VirtualAlloc()`` on Windows, they are discontiguous. Releasing a memory mapping gives back the memory immediatly to the system. For the heap, memory is only gave back to the system if it is at the end of the heap. Otherwise, the memory will only gave back to the system when all the memory located after the released memory are also released. This limitation causes an issue called the "memory fragmentation": the memory usage seen by the system may be much higher than real usage. Windows provides a `Low-fragmentation Heap `_. The Linux kernel uses `slab allocation `_. The glib library has a `Memory Slice API `_: efficient way to allocate groups of equal-sized chunks of memory Links ===== CPython issues related to memory allocation: * `Issue #3329: Add new APIs to customize memory allocators `_ * `Issue #13483: Use VirtualAlloc to allocate memory arenas `_ * `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which isn't thread safe `_ * `Issue #18203: Replace calls to malloc() with PyMem_Malloc() `_ * `Issue #18227: Use Python memory allocators in external libraries like zlib or OpenSSL `_ Projects analyzing the memory usage of Python applications: * `pytracemalloc `_ * `Meliae: Python Memory Usage Analyzer `_ * `Guppy-PE: umbrella package combining Heapy and GSL `_ * `PySizer (developed for Python 2.4) `_