python-peps/pep-0445.txt

PEP: 445
Title: Add new APIs to customize Python memory allocators
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 15-june-2013
Python-Version: 3.4

Abstract
========

Add new APIs to customize Python memory allocators.


Rationale
=========

Use cases:

* Application embedding Python may want to isolate Python memory from
  the memory of the application, or may want to use a different memory
  allocator optimized for its Python usage
* Python running on embedded devices with low memory and slow CPU.
  A custom memory allocator may be required to use efficiently the
  memory and/or to be able to use all the memory of the device.
* Debug tool to:

  - track the memory usage (memory leaks)
  - get the Python filename and line number where an object was
    allocated
  - detect buffer underflow, buffer overflow and detect misuse of Python
    allocator APIs (builtin Python debug hooks)
  - force allocation to fail to test handling of ``MemoryError``
    exception


Proposal
========

New functions and new structure
-------------------------------

* Add a new GIL-free (no need to hold the GIL) memory allocator:

  - ``void* PyMem_RawMalloc(size_t size)``
  - ``void* PyMem_RawRealloc(void *ptr, size_t new_size)``
  - ``void PyMem_RawFree(void *ptr)``
  - The newly allocated memory will not have been initialized in any
    way.
  - Requesting zero bytes returns a distinct non-*NULL* pointer if
    possible, as if ``PyMem_Malloc(1)`` had been called instead.

* Add a new ``PyMemAllocator`` structure::

    typedef struct {
        /* user context passed as the first argument
           to the 3 functions */
        void *ctx;

        /* allocate a memory block */
        void* (*malloc) (void *ctx, size_t size);

        /* allocate or resize a memory block */
        void* (*realloc) (void *ctx, void *ptr, size_t new_size);

        /* release a memory block */
        void (*free) (void *ctx, void *ptr);
    } PyMemAllocator;

* Add a new ``PyMemAllocatorDomain`` enum to choose the Python
  allocator domain. Domains:

  - ``PYMEM_DOMAIN_RAW``: ``PyMem_RawMalloc()``, ``PyMem_RawRealloc()``
    and ``PyMem_RawFree()``

  - ``PYMEM_DOMAIN_MEM``: ``PyMem_Malloc()``, ``PyMem_Realloc()`` and
    ``PyMem_Free()``

  - ``PYMEM_DOMAIN_OBJ``: ``PyObject_Malloc()``, ``PyObject_Realloc()``
    and ``PyObject_Free()``

* Add new functions to get and set memory allocators:

  - ``void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)``
  - ``void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)``
  - The new allocator must return a distinct non-*NULL* pointer when
    requesting zero bytes

* Add a new ``PyObjectArenaAllocator`` structure::

    typedef struct {
        /* user context passed as the first argument
           to the 2 functions */
        void *ctx;

        /* allocate an arena */
        void* (*alloc) (void *ctx, size_t size);

        /* release an arena */
        void (*free) (void *ctx, void *ptr, size_t size);
    } PyObjectArenaAllocator;

* Add new functions to get and set the arena allocator used by
  *pymalloc*:

  - ``void PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)``
  - ``void PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)``

* Add a new function to setup the builtin Python debug hooks when a
  memory allocator is replaced:

  - ``void PyMem_SetupDebugHooks(void)``
  - the function does nothing is Python is not compiled in debug mode

* Memory allocators always returns *NULL* if size is greater than
  ``PY_SSIZE_T_MAX``. The check is done before calling the
  inner function.

The *pymalloc* allocator is optimized for objects smaller than 512 bytes
with a short lifetime. It uses memory mappings with a fixed size of 256
KB called "arenas".

Default allocators:

* ``PYMEM_DOMAIN_RAW``, ``PYMEM_DOMAIN_MEM``: ``malloc()``,
  ``realloc()``, ``free()`` (and *ctx* is NULL); call ``malloc(1)`` when
  requesting zero bytes
* ``PYMEM_DOMAIN_OBJ``: *pymalloc* allocator which fall backs on
  ``PyMem_Malloc()`` for allocations larger than 512 bytes
* *pymalloc* arena allocator: ``mmap()``, ``munmap()`` (and *ctx* is
  NULL), or ``malloc()`` and ``free()`` if ``mmap()`` is not available

The builtin Python debug hooks were introduced in Python 2.3 and
implement the following checks:

* Newly allocated memory is filled with the byte ``0xCB``, freed memory
  is filled with the byte ``0xDB``.
* Detect API violations, ex: ``PyObject_Free()`` called on a memory
  block allocated by ``PyMem_Malloc()``
* Detect write before the start of the buffer (buffer underflow)
* Detect write after the end of the buffer (buffer overflow)


Don't call malloc() directly anymore
------------------------------------

``PyMem_Malloc()`` and ``PyMem_Realloc()`` always call ``malloc()`` and
``realloc()``, instead of calling ``PyObject_Malloc()`` and
``PyObject_Realloc()`` in debug mode.

``PyObject_Malloc()`` falls back on ``PyMem_Malloc()`` instead of
``malloc()`` if size is greater or equal than 512 bytes, and
``PyObject_Realloc()`` falls back on ``PyMem_Realloc()`` instead of
``realloc()``

Replace direct calls to ``malloc()`` with ``PyMem_Malloc()``, or
``PyMem_RawMalloc()`` if the GIL is not held.

Configure external libraries like zlib or OpenSSL to allocate memory
using ``PyMem_Malloc()`` or ``PyMem_RawMalloc()``. If the allocator of a
library can only be replaced globally, the allocator is not replaced if
Python is embedded in an application.

For the "track memory usage" use case, it is important to track memory
allocated in external libraries to have accurate reports, because these
allocations may be large.

If an hook is used to the track memory usage, the memory allocated by
``malloc()`` will not be tracked. Remaining ``malloc()`` in external
libraries like OpenSSL or bz2 may allocate large memory blocks and so
would be missed in memory usage reports.


Examples
========

Use case 1: Replace Memory Allocator, keep pymalloc
----------------------------------------------------

Dummy example wasting 2 bytes per memory block,
and 10 bytes per memory mapping::

    #include <stdlib.h>

    int alloc_padding = 2;
    int arena_padding = 10;

    void* my_malloc(void *ctx, size_t size)
    {
        int padding = *(int *)ctx;
        return malloc(size + padding);
    }

    void* my_realloc(void *ctx, void *ptr, size_t new_size)
    {
        int padding = *(int *)ctx;
        return realloc(ptr, new_size + padding);
    }

    void my_free(void *ctx, void *ptr)
    {
        free(ptr);
    }

    void* my_alloc_arena(void *ctx, size_t size)
    {
        int padding = *(int *)ctx;
        return malloc(size + padding);
    }

    void my_free_arena(void *ctx, void *ptr, size_t size)
    {
        free(ptr);
    }

    void setup_custom_allocator(void)
    {
        PyMemAllocator alloc;
        PyObjectArenaAllocator arena;

        alloc.ctx = &alloc_padding;
        alloc.malloc = my_malloc;
        alloc.realloc = my_realloc;
        alloc.free = my_free;

        PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc);
        PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &alloc);

        arena.ctx = &arena_padding;
        arena.alloc = my_alloc_arena;
        arena.free = my_free_arena;
        PyObject_SetArenaAllocator(&arena);

        PyMem_SetupDebugHooks();
    }

.. warning::
   Remove the call ``PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc)`` if
   the new allocator is not thread-safe.


Use case 2: Replace Memory Allocator, override pymalloc
--------------------------------------------------------

If your allocator is optimized for allocations of objects smaller than
512 bytes with a short lifetime, pymalloc can be overriden (replace
``PyObject_Malloc()``).

Dummy example wasting 2 bytes per memory block::

    #include <stdlib.h>

    int padding = 2;

    void* my_malloc(void *ctx, size_t size)
    {
        int padding = *(int *)ctx;
        return malloc(size + padding);
    }

    void* my_realloc(void *ctx, void *ptr, size_t new_size)
    {
        int padding = *(int *)ctx;
        return realloc(ptr, new_size + padding);
    }

    void my_free(void *ctx, void *ptr)
    {
        free(ptr);
    }

    void setup_custom_allocator(void)
    {
        PyMemAllocator alloc;
        alloc.ctx = &padding;
        alloc.malloc = my_malloc;
        alloc.realloc = my_realloc;
        alloc.free = my_free;

        PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc);
        PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &alloc);
        PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &alloc);

        PyMem_SetupDebugHooks();
    }

.. warning::
   Remove the call ``PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc)`` if
   the new allocator is not thread-safe.



Use case 3: Setup Allocator Hooks
---------------------------------

Example to setup hooks on all memory allocators::

    struct {
        PyMemAllocator raw;
        PyMemAllocator mem;
        PyMemAllocator obj;
        /* ... */
    } hook;

    static void* hook_malloc(void *ctx, size_t size)
    {
        PyMemAllocator *alloc = (PyMemAllocator *)ctx;
        /* ... */
        ptr = alloc->malloc(alloc->ctx, size);
        /* ... */
        return ptr;
    }

    static void* hook_realloc(void *ctx, void *ptr, size_t new_size)
    {
        PyMemAllocator *alloc = (PyMemAllocator *)ctx;
        void *ptr2;
        /* ... */
        ptr2 = alloc->realloc(alloc->ctx, ptr, new_size);
        /* ... */
        return ptr2;
    }

    static void hook_free(void *ctx, void *ptr)
    {
        PyMemAllocator *alloc = (PyMemAllocator *)ctx;
        /* ... */
        alloc->free(alloc->ctx, ptr);
        /* ... */
    }

    void setup_hooks(void)
    {
        PyMemAllocator alloc;
        static int installed = 0;

        if (installed)
            return;
        installed = 1;

        alloc.malloc = hook_malloc;
        alloc.realloc = hook_realloc;
        alloc.free = hook_free;
        PyMem_GetAllocator(PYMEM_DOMAIN_RAW, &hook.raw);
        PyMem_GetAllocator(PYMEM_DOMAIN_MEM, &hook.mem);
        PyMem_GetAllocator(PYMEM_DOMAIN_OBJ, &hook.obj);

        alloc.ctx = &hook.raw;
        PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc);

        alloc.ctx = &hook.mem;
        PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &alloc);

        alloc.ctx = &hook.obj;
        PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &alloc);
    }

.. warning::
   Remove the call ``PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc)`` if
   hooks are not thread-safe.

.. note::
   ``PyMem_SetupDebugHooks()`` does not need to be called because the
   allocator is not replaced: Python debug hooks are installed
   automatically at startup.


Performances
============

Results of the `Python benchmarks suite
<http://hg.python.org/benchmarks>`_ (-b 2n3): some tests are 1.04x
faster, some tests are 1.04 slower, significant is between 115 and -191.

Results of pybench benchmark: "+0.1%" slower globally (diff between
-4.9% and +5.6%).

The full reports are attached to the issue #3329.


Alternatives
============

More specific functions to get/set memory allocators
----------------------------------------------------

Replace the 2 functions:

* ``void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)``
* ``void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)``

with:

* ``void PyMem_GetRawAllocator(PyMemAllocator *allocator)``
* ``void PyMem_GetAllocator(PyMemAllocator *allocator)``
* ``void PyObject_GetAllocator(PyMemAllocator *allocator)``
* ``void PyMem_SetRawAllocator(PyMemAllocator *allocator)``
* ``void PyMem_SetAllocator(PyMemAllocator *allocator)``
* ``void PyObject_SetAllocator(PyMemAllocator *allocator)``


Make PyMem_Malloc() reuse PyMem_RawMalloc() by default
------------------------------------------------------

``PyMem_Malloc()`` should call ``PyMem_RawMalloc()`` by default. So
calling ``PyMem_SetRawAllocator()`` would also also patch
``PyMem_Malloc()`` indirectly.


Add a new PYDEBUGMALLOC environment variable
--------------------------------------------

To be able to use the Python builtin debug hooks even when a custom
memory allocator replaces the default Python allocator, an environment
variable ``PYDEBUGMALLOC`` can be added to setup these debug function
hooks, instead of adding the new function ``PyMem_SetupDebugHooks()``.
If the environment variable is present, ``PyMem_SetRawAllocator()``,
``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()`` will reinstall
automatically the hook on top of the new allocator.

An new environment variable would make the Python initialization even
more complex. The `PEP 432 <http://www.python.org/dev/peps/pep-0432/>`_
tries to simply the CPython startup sequence.


Use macros to get customizable allocators
-----------------------------------------

To have no overhead in the default configuration, customizable
allocators would be an optional feature enabled by a configuration
option or by macros.

Not having to recompile Python makes debug hooks easier to use in
practice.  Extensions modules don't have to be recompiled with macros.


Pass the C filename and line number
-----------------------------------

Define allocator functions as macros using ``__FILE__`` and ``__LINE__``
to get the C filename and line number of a memory allocation.

Example of ``PyMem_Malloc`` macro with the modified
``PyMemAllocator`` structure::

    typedef struct {
        /* user context passed as the first argument
           to the 3 functions */
        void *ctx;

        /* allocate a memory block */
        void* (*malloc) (void *ctx, const char *filename, int lineno,
                         size_t size);

        /* allocate or resize a memory block */
        void* (*realloc) (void *ctx, const char *filename, int lineno,
                          void *ptr, size_t new_size);

        /* release a memory block */
        void (*free) (void *ctx, const char *filename, int lineno,
                      void *ptr);
    } PyMemAllocator;

    void* _PyMem_MallocTrace(const char *filename, int lineno,
                             size_t size);

    /* need also a function for the Python stable ABI */
    void* PyMem_Malloc(size_t size);

    #define PyMem_Malloc(size) \
            _PyMem_MallocTrace(__FILE__, __LINE__, size)

Passing a filename and a line number to each allocator makes the API more
complex: pass 3 new arguments, instead of just a context argument, to each
allocator function. The GC allocator functions should also be patched.
For example, ``_PyObject_GC_Malloc()`` is used in many C functions and so
objects of differenet types would have the same allocation location. Such
changes add too much complexity for a little gain.


GIL-free PyMem_Malloc()
-----------------------

In Python 3.3, when Python is compiled in debug mode, ``PyMem_Malloc()``
calls indirectly ``PyObject_Malloc()`` which requires the GIL to be
held.  That's why ``PyMem_Malloc()`` must be called with the GIL held.

This PEP proposes changes ``PyMem_Malloc()``: it now always call
``malloc()``.  The "GIL must be held" restriction can be removed from
``PyMem_Malloc()``.

Allowing to call ``PyMem_Malloc()`` without holding the GIL might break
applications which setup their own allocators or allocator hooks.
Holding the GIL is convinient to develop a custom allocator: no need to
care of other threads. It is also convinient for a debug allocator hook:
Python internal objects can be safetly inspected.

Calling ``PyGILState_Ensure()`` in a memory allocator may have
unexpected behaviour, especially at Python startup and at creation of a
new Python thread state.


Don't add PyMem_RawMalloc()
---------------------------

Replace ``malloc()`` with ``PyMem_Malloc()``, but only if the GIL is
held.  Otherwise, keep ``malloc()`` unchanged.

The ``PyMem_Malloc()`` is used without the GIL held in some Python
functions.  For example, the ``main()`` and ``Py_Main()`` functions of
Python call ``PyMem_Malloc()`` whereas the GIL do not exist yet. In this
case, ``PyMem_Malloc()`` should be replaced with ``malloc()`` (or
``PyMem_RawMalloc()``).

If an hook is used to the track memory usage, the memory allocated by
direct calls to ``malloc()`` will not be tracked. External libraries
like OpenSSL or bz2 should not call ``malloc()`` directly, so large
allocated will be included in memory usage reports.


Use existing debug tools to analyze the memory
----------------------------------------------

There are many existing debug tools to analyze the memory. Some
examples: `Valgrind <http://valgrind.org/>`_, `Purify
<http://ibm.com/software/awdtools/purify/>`_, `Clang AddressSanitizer
<http://code.google.com/p/address-sanitizer/>`_, `failmalloc
<http://www.nongnu.org/failmalloc/>`_, etc.

The problem is to retrieve the Python object related to a memory pointer
to read its type and/or content. Another issue is to retrieve the
location of the memory allocation: the C backtrace is usually useless
(same reasoning than macros using ``__FILE__`` and ``__LINE__``), the
Python filename and line number (or even the Python traceback) is more
useful.

Classic tools are unable to introspect Python internals to collect such
information. Being able to setup a hook on allocators called with the
GIL held allow to collect a lot of useful data from Python internals.


Add msize()
-----------

Add another field to ``PyMemAllocator`` and ``PyObjectArenaAllocator``::

    size_t msize(void *ptr);

This function returns the size of a memory block or a memory mapping.
Return (size_t)-1 if the function is not implemented or if the pointer
is unknown (ex: NULL pointer).

On Windows, this function can be implemented using ``_msize()`` and
``VirtualQuery()``.


No context argument
-------------------

Simplify the signature of allocator functions, remove the context
argument:

* ``void* malloc(size_t size)``
* ``void* realloc(void *ptr, size_t new_size)``
* ``void free(void *ptr)``

It is likely for an allocator hook to be reused for
``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()``, or even
``PyMem_SetRawAllocator()``, but the hook must call a different function
depending on the allocator. The context is a convenient way to reuse the
same custom allocator or hook for different Python allocators.

In C++, the context can be used to pass *this*.


External libraries
==================

Python should try to reuse the same prototypes for allocator functions
than other libraries.

Libraries used by Python:

* OpenSSL: `CRYPTO_set_mem_functions()
  <http://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=crypto/mem.c;h=f7984fa958eb1edd6c61f6667f3f2b29753be662;hb=HEAD#l124>`_
  to set memory management functions globally
* expat: `parserCreate()
  <http://hg.python.org/cpython/file/cc27d50bd91a/Modules/expat/xmlparse.c#l724>`_
  has a per-instance memory handler
* zlib: `zlib 1.2.8 Manual <http://www.zlib.net/manual.html#Usage>`_,
  pass an opaque pointer
* bz2: `bzip2 and libbzip2, version 1.0.5
  <http://www.bzip.org/1.0.5/bzip2-manual-1.0.5.html>`_,
  pass an opaque pointer
* lzma: `LZMA SDK - How to Use
  <http://www.asawicki.info/news_1368_lzma_sdk_-_how_to_use.html>`_,
  pass an opaque pointer
* lipmpdec doesn't have this extra *ctx* parameter

Other libraries:

* glib: `g_mem_set_vtable()
  <http://developer.gnome.org/glib/unstable/glib-Memory-Allocation.html#g-mem-set-vtable>`_
* libxml2: `xmlGcMemSetup() <http://xmlsoft.org/html/libxml-xmlmemory.html>`_,
  global
* Oracle's OCI: `Oracle Call Interface Programmer's Guide,
  Release 2 (9.2)
  <http://docs.oracle.com/cd/B10501_01/appdev.920/a96584/oci15re4.htm>`_

See also the `GNU libc: Memory Allocation Hooks
<http://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html>`_.


Memory allocators
=================

The C standard library provides the well known ``malloc()`` function.
Its implementation depends on the platform and of the C library. The GNU
C library uses a modified ptmalloc2, based on "Doug Lea's Malloc"
(dlmalloc). FreeBSD uses `jemalloc
<http://www.canonware.com/jemalloc/>`_. Google provides tcmalloc which
is part of `gperftools <http://code.google.com/p/gperftools/>`_.

``malloc()`` uses two kinds of memory: heap and memory mappings. Memory
mappings are usually used for large allocations (ex: larger than 256
KB), whereas the heap is used for small allocations.

On UNIX, the heap is handled by ``brk()`` and ``sbrk()`` system calls on
Linux, and it is contiguous.  On Windows, the heap is handled by
``HeapAlloc()`` and may be discontiguous. Memory mappings are handled by
``mmap()`` on UNIX and ``VirtualAlloc()`` on Windows, they may be
discontiguous.

Releasing a memory mapping gives back immediatly the memory to the
system. On UNIX, heap memory is only given back to the system if it is
at the end of the heap. Otherwise, the memory will only be given back to
the system when all the memory located after the released memory are
also released.

To allocate memory in the heap, the allocator tries to reuse free space.
If there is no contiguous space big enough, the heap must be increased,
even if we have more free space than required size.  This issue is
called the "memory fragmentation": the memory usage seen by the system
may be much higher than real usage. On Windows, ``HeapAlloc()`` creates
a new memory mapping with ``VirtualAlloc()`` if there is not enough free
contiguous memory.

CPython has a *pymalloc* allocator for allocations smaller than 512
bytes. This allocator is optimized for small objects with a short
lifetime. It uses memory mappings called "arenas" with a fixed size of
256 KB.

Other allocators:

* Windows provides a `Low-fragmentation Heap
  <http://msdn.microsoft.com/en-us/library/windows/desktop/aa366750%28v=vs.85%29.aspx>`_.

* The Linux kernel uses `slab allocation
  <http://en.wikipedia.org/wiki/Slab_allocation>`_.

* The glib library has a `Memory Slice API
  <https://developer.gnome.org/glib/unstable/glib-Memory-Slices.html>`_:
  efficient way to allocate groups of equal-sized chunks of memory


Links
=====

CPython issues related to memory allocation:

* `Issue #3329: Add new APIs to customize memory allocators
  <http://bugs.python.org/issue3329>`_
* `Issue #13483: Use VirtualAlloc to allocate memory arenas
  <http://bugs.python.org/issue13483>`_
* `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which
  isn't thread safe <http://bugs.python.org/issue16742>`_
* `Issue #18203: Replace calls to malloc() with PyMem_Malloc() or
  PyMem_RawMalloc() <http://bugs.python.org/issue18203>`_
* `Issue #18227: Use Python memory allocators in external libraries like
  zlib or OpenSSL <http://bugs.python.org/issue18227>`_

Projects analyzing the memory usage of Python applications:

* `pytracemalloc
  <https://pypi.python.org/pypi/pytracemalloc>`_
* `Meliae: Python Memory Usage Analyzer
  <https://pypi.python.org/pypi/meliae>`_
* `Guppy-PE: umbrella package combining Heapy and GSL
  <http://guppy-pe.sourceforge.net/>`_
* `PySizer (developed for Python 2.4)
  <http://pysizer.8325.org/>`_