diff --git a/pep-0445.txt b/pep-0445.txt index 1e158c37e..98178990e 100644 --- a/pep-0445.txt +++ b/pep-0445.txt @@ -151,13 +151,65 @@ CCP API XXX To be done (Kristján Valur Jónsson) XXX +External libraries +================== + +* glib: `g_mem_set_vtable() + `_ + + +Memory allocators +================= + +The C standard library provides the well known ``malloc()`` function. Its +implementation depends on the platform and of the C library. The GNU C library +uses a modified ptmalloc2, based on "Doug Lea's Malloc" (dlmalloc). FreeBSD +uses `jemalloc `_. Google provides +tcmalloc which is part of `gperftools `_. + +``malloc()`` uses two kinds of memory: heap and memory mappings. Memory +mappings are usually used for large allocations (ex: larger than 256 KB), +whereas the heap is used for small allocations. + +The heap is handled by ``brk()`` and ``sbrk()`` system calls on Linux, and is +contiguous. Memory mappings are handled by ``mmap()`` on UNIX and +``VirtualAlloc()`` on Windows, they are discontiguous. Releasing a memory +mapping gives back the memory immediatly to the system. For the heap, memory is +only gave back to the system if it is at the end of the heap. Otherwise, the +memory will only gave back to the system when all the memory located after the +released memory are also released. This limitation causes an issue called the +"memory fragmentation": the memory usage seen by the system may be much higher +than real usage. + +Windows provides a `Low-fragmentation Heap +`_. + +The Linux kernel uses `slab allocation +`_. + +The glib library has a `Memory Slice API +`_: +efficient way to allocate groups of equal-sized chunks of memory + + Links ===== -Memory allocators: +CPython issues related to memory allocation: * `Issue #3329: Add new APIs to customize memory allocators `_ +* `Issue #13483: Use VirtualAlloc to allocate memory arenas + `_ +* `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which + isn't thread safe `_ +* `Issue #18203: Replace calls to malloc() with PyMem_Malloc() + `_ +* `Issue #18227: Use Python memory allocators in external libraries like zlib + or OpenSSL `_ + +Projects analyzing the memory usage of Python applications: + * `pytracemalloc `_ * `Meliae: Python Memory Usage Analyzer @@ -167,6 +219,11 @@ Memory allocators: * `PySizer (developed for Python 2.4) `_ +APIs to set a custom memory allocator and/or hook memory allocators: + +* `GNU libc: Memory Allocation Hooks + `_ + Other: * `Python benchmark suite