iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

PEP 445: textwidth=72 (for email)

This commit is contained in:
Victor Stinner 2013-06-18 22:05:17 +02:00
parent 2d81cffb61
commit d1c9cb312b
1 changed files with 163 additions and 145 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

308
pep-0445.txt
View File

@ -20,19 +20,21 @@ Rationale
Use cases: Use cases:
* Application embedding Python may want to isolate Python memory from the * Application embedding Python may want to isolate Python memory from
memory of the application, or may want to different memory allocator the memory of the application, or may want to different memory
optimized for its Python usage allocator optimized for its Python usage
* Python running on embedded devices with low memory and slow CPU. * Python running on embedded devices with low memory and slow CPU.
A custom memory allocator may be required to use efficiently the memory A custom memory allocator may be required to use efficiently the
and/or to be able to use all the memory of the device. memory and/or to be able to use all the memory of the device.
* Debug tool to: * Debug tool to:
- track memory usage (memory leaks) - track memory usage (memory leaks)
- get the Python filename and line number where an object was allocated - get the Python filename and line number where an object was
allocated
- detect buffer underflow, buffer overflow and detect misuse of Python - detect buffer underflow, buffer overflow and detect misuse of Python
allocator APIs (builtin Python debug hooks) allocator APIs (builtin Python debug hooks)
- force allocation to fail to test handling of ``MemoryError`` exception - force allocation to fail to test handling of ``MemoryError``
exception
Proposal Proposal
@ -46,13 +48,14 @@ API changes
- ``void* PyMem_RawMalloc(size_t size)`` - ``void* PyMem_RawMalloc(size_t size)``
- ``void* PyMem_RawRealloc(void *ptr, size_t new_size)`` - ``void* PyMem_RawRealloc(void *ptr, size_t new_size)``
- ``void PyMem_RawFree(void *ptr)`` - ``void PyMem_RawFree(void *ptr)``
- the behaviour of requesting zero bytes is not defined: return *NULL* or a - the behaviour of requesting zero bytes is not defined: return *NULL*
distinct non-*NULL* pointer depending on the platform. or a distinct non-*NULL* pointer depending on the platform.
* Add a new ``PyMemBlockAllocator`` structure:: * Add a new ``PyMemBlockAllocator`` structure::
typedef struct { typedef struct {
/* user context passed as the first argument to the 3 functions */ /* user context passed as the first argument
to the 3 functions */
void *ctx; void *ctx;
/* allocate a memory block */ /* allocate a memory block */
@ -65,32 +68,34 @@ API changes
void (*free) (void *ctx, void *ptr); void (*free) (void *ctx, void *ptr);
} PyMemBlockAllocator; } PyMemBlockAllocator;
* Add new functions to get and set internal functions of ``PyMem_RawMalloc()``, * Add new functions to get and set internal functions of
``PyMem_RawRealloc()`` and ``PyMem_RawFree()``: ``PyMem_RawMalloc()``, ``PyMem_RawRealloc()`` and ``PyMem_RawFree()``:
- ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)`` - ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)``
- ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)`` - ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)``
* Add new functions to get and set internal functions of ``PyMem_Malloc()``, * Add new functions to get and set internal functions of
``PyMem_Realloc()`` and ``PyMem_Free()``: ``PyMem_Malloc()``, ``PyMem_Realloc()`` and ``PyMem_Free()``:
- ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)`` - ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)``
- ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)`` - ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)``
- ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return *NULL*: - ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return
it would be treated as an error. *NULL*: it would be treated as an error.
* Add new functions to get and set internal functions of * Add new functions to get and set internal functions of
``PyObject_Malloc()``,, ``PyObject_Realloc()`` and ``PyObject_Free()``: ``PyObject_Malloc()``,, ``PyObject_Realloc()`` and
``PyObject_Free()``:
- ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)`` - ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)``
- ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)`` - ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)``
- ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return *NULL*: - ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return
it would be treated as an error. *NULL*: it would be treated as an error.
* Add a new ``PyMemMappingAllocator`` structure:: * Add a new ``PyMemMappingAllocator`` structure::
typedef struct { typedef struct {
/* user context passed as the first argument to the 2 functions */ /* user context passed as the first argument
to the 2 functions */
void *ctx; void *ctx;
/* allocate a memory mapping */ /* allocate a memory mapping */
@ -104,26 +109,26 @@ API changes
- ``void PyMem_GetMappingAllocator(PyMemMappingAllocator *allocator)`` - ``void PyMem_GetMappingAllocator(PyMemMappingAllocator *allocator)``
- ``void PyMem_SetMappingAllocator(PyMemMappingAllocator *allocator)`` - ``void PyMem_SetMappingAllocator(PyMemMappingAllocator *allocator)``
- Currently, this allocator is only used internally by *pymalloc* to allocate - Currently, this allocator is only used internally by *pymalloc* to
arenas. allocate arenas.
* Add a new function to setup the builtin Python debug hooks when memory * Add a new function to setup the builtin Python debug hooks when memory
allocators are replaced: allocators are replaced:
- ``void PyMem_SetupDebugHooks(void)`` - ``void PyMem_SetupDebugHooks(void)``
* The following memory allocators always returns *NULL* if size is greater * The following memory allocators always returns *NULL* if size is
than ``PY_SSIZE_T_MAX``: ``PyMem_RawMalloc()``, ``PyMem_RawRealloc()``, greater than ``PY_SSIZE_T_MAX``: ``PyMem_RawMalloc()``,
``PyMem_Malloc()``, ``PyMem_Realloc()``, ``PyObject_Malloc()``, ``PyMem_RawRealloc()``, ``PyMem_Malloc()``, ``PyMem_Realloc()``,
``PyObject_Realloc()``. ``PyObject_Malloc()``, ``PyObject_Realloc()``.
The builtin Python debug hooks were introduced in Python 2.3 and implement the The builtin Python debug hooks were introduced in Python 2.3 and
following checks: implement the following checks:
* Newly allocated memory is filled with the byte ``0xCB``, freed memory is * Newly allocated memory is filled with the byte ``0xCB``, freed memory
filled with the byte ``0xDB``. is filled with the byte ``0xDB``.
* Detect API violations, ex: ``PyObject_Free()`` called on a memory block * Detect API violations, ex: ``PyObject_Free()`` called on a memory
allocated by ``PyMem_Malloc()`` block allocated by ``PyMem_Malloc()``
* Detect write before the start of the buffer (buffer underflow) * Detect write before the start of the buffer (buffer underflow)
* Detect write after the end of the buffer (buffer overflow) * Detect write after the end of the buffer (buffer overflow)
@ -134,20 +139,20 @@ The *pymalloc* allocator is used by default for:
Make usage of these new APIs Make usage of these new APIs
---------------------------- ----------------------------
* ``PyMem_Malloc()`` and ``PyMem_Realloc()`` always call ``malloc()`` and * ``PyMem_Malloc()`` and ``PyMem_Realloc()`` always call ``malloc()``
``realloc()``, instead of calling ``PyObject_Malloc()`` and and ``realloc()``, instead of calling ``PyObject_Malloc()`` and
``PyObject_Realloc()`` in debug mode ``PyObject_Realloc()`` in debug mode
* ``PyObject_Malloc()`` falls back on ``PyMem_Malloc()`` instead of * ``PyObject_Malloc()`` falls back on ``PyMem_Malloc()`` instead of
``malloc()`` if size is greater or equal than ``SMALL_REQUEST_THRESHOLD`` ``malloc()`` if size is greater or equal than
(512 bytes), and ``PyObject_Realloc()`` falls back on ``PyMem_Realloc()`` ``SMALL_REQUEST_THRESHOLD`` (512 bytes), and ``PyObject_Realloc()``
instead of ``realloc()`` falls back on ``PyMem_Realloc()`` instead of ``realloc()``
* Replace direct calls to ``malloc()`` with ``PyMem_Malloc()``, or * Replace direct calls to ``malloc()`` with ``PyMem_Malloc()``, or
``PyMem_RawMalloc()`` if the GIL is not held ``PyMem_RawMalloc()`` if the GIL is not held
* Configure external libraries like zlib or OpenSSL to allocate memory using * Configure external libraries like zlib or OpenSSL to allocate memory
``PyMem_RawMalloc()`` using ``PyMem_RawMalloc()``
Examples Examples
@ -213,16 +218,16 @@ Dummy example wasting 2 bytes per allocation, and 10 bytes per arena::
} }
.. warning:: .. warning::
Remove the call ``PyMem_SetRawAllocator(&alloc)`` if the new allocator Remove the call ``PyMem_SetRawAllocator(&alloc)`` if the new
are not thread-safe. allocator are not thread-safe.
Use case 2: Replace Memory Allocator, override pymalloc Use case 2: Replace Memory Allocator, override pymalloc
-------------------------------------------------------- --------------------------------------------------------
If your allocator is optimized for allocation of small objects (less than 512 If your allocator is optimized for allocation of small objects (less
bytes) with a short lifetime, pymalloc can be overriden: replace than 512 bytes) with a short lifetime, pymalloc can be overriden:
``PyObject_Malloc()``. replace ``PyObject_Malloc()``.
Dummy Example wasting 2 bytes per allocation:: Dummy Example wasting 2 bytes per allocation::
@ -263,8 +268,8 @@ Dummy Example wasting 2 bytes per allocation::
} }
.. warning:: .. warning::
Remove the call ``PyMem_SetRawAllocator(&alloc)`` if the new allocator Remove the call ``PyMem_SetRawAllocator(&alloc)`` if the new
are not thread-safe. allocator are not thread-safe.
@ -338,20 +343,21 @@ Example to setup hooks on all memory allocators::
thread-safe. thread-safe.
.. note:: .. note::
``PyMem_SetupDebugHooks()`` does not need to be called: Python debug hooks ``PyMem_SetupDebugHooks()`` does not need to be called: Python debug
are installed automatically at startup. hooks are installed automatically at startup.
Performances Performances
============ ============
Results of the `Python benchmarks suite <http://hg.python.org/benchmarks>`_ (-b Results of the `Python benchmarks suite
2n3): some tests are 1.04x faster, some tests are 1.04 slower, significant is <http://hg.python.org/benchmarks>`_ (-b 2n3): some tests are 1.04x
between 115 and -191. I don't understand these output, but I guess that the faster, some tests are 1.04 slower, significant is between 115 and -191.
overhead cannot be seen with such test. I don't understand these output, but I guess that the overhead cannot be
seen with such test.
Results of pybench benchmark: "+0.1%" slower globally (diff between -4.9% and Results of pybench benchmark: "+0.1%" slower globally (diff between
+5.6%). -4.9% and +5.6%).
The full reports are attached to the issue #3329. The full reports are attached to the issue #3329.
@ -390,9 +396,9 @@ Drawback: the caller has to check if the result is 0, or handle the error.
PyMem_Malloc() reuses PyMem_RawMalloc() by default PyMem_Malloc() reuses PyMem_RawMalloc() by default
-------------------------------------------------- --------------------------------------------------
``PyMem_Malloc()`` should call ``PyMem_RawMalloc()`` by default. So calling ``PyMem_Malloc()`` should call ``PyMem_RawMalloc()`` by default. So
``PyMem_SetRawAllocator()`` would also also patch ``PyMem_Malloc()`` calling ``PyMem_SetRawAllocator()`` would also also patch
indirectly. ``PyMem_Malloc()`` indirectly.
.. note:: .. note::
@ -404,39 +410,42 @@ indirectly.
Add a new PYDEBUGMALLOC environment variable Add a new PYDEBUGMALLOC environment variable
-------------------------------------------- --------------------------------------------
To be able to use the Python builtin debug hooks even when a custom memory To be able to use the Python builtin debug hooks even when a custom
allocator replaces the default Python allocator, an environment variable memory allocator replaces the default Python allocator, an environment
``PYDEBUGMALLOC`` can be added to setup these debug function hooks, instead of variable ``PYDEBUGMALLOC`` can be added to setup these debug function
adding the new function ``PyMem_SetupDebugHooks()``. If the environment hooks, instead of adding the new function ``PyMem_SetupDebugHooks()``.
variable is present, ``PyMem_SetRawAllocator()``, ``PyMem_SetAllocator()`` If the environment variable is present, ``PyMem_SetRawAllocator()``,
and ``PyObject_SetAllocator()`` will reinstall automatically the hook on top ``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()`` will reinstall
of the new allocator. automatically the hook on top of the new allocator.
An new environment variable would make the Python initialization even more An new environment variable would make the Python initialization even
complex. The `PEP 432 <http://www.python.org/dev/peps/pep-0432/>`_ tries to more complex. The `PEP 432 <http://www.python.org/dev/peps/pep-0432/>`_
simply the CPython startup sequence. tries to simply the CPython startup sequence.
Use macros to get customizable allocators Use macros to get customizable allocators
----------------------------------------- -----------------------------------------
To have no overhead in the default configuration, customizable allocators would To have no overhead in the default configuration, customizable
be an optional feature enabled by a configuration option or by macros. allocators would be an optional feature enabled by a configuration
option or by macros.
Not having to recompile Python makes debug hooks easier to use in practice. Not having to recompile Python makes debug hooks easier to use in
Extensions modules don't have to be recompiled with macros. practice. Extensions modules don't have to be recompiled with macros.
Pass the C filename and line number Pass the C filename and line number
----------------------------------- -----------------------------------
Define allocator functions using macros and use ``__FILE__`` and ``__LINE__`` Define allocator functions using macros and use ``__FILE__`` and
to get the C filename and line number of a memory allocation. ``__LINE__`` to get the C filename and line number of a memory
allocation.
Example:: Example::
typedef struct { typedef struct {
/* user context passed as the first argument to the 3 functions */ /* user context passed as the first argument
to the 3 functions */
void *ctx; void *ctx;
/* allocate a memory block */ /* allocate a memory block */
@ -452,7 +461,8 @@ Example::
void *ptr); void *ptr);
} PyMemBlockAllocator; } PyMemBlockAllocator;
void* _PyMem_MallocTrace(const char *filename, int lineno, size_t size); void* _PyMem_MallocTrace(const char *filename, int lineno,
size_t size);
/* need also a function for the Python stable ABI */ /* need also a function for the Python stable ABI */
void* PyMem_Malloc(size_t size); void* PyMem_Malloc(size_t size);
@ -470,7 +480,8 @@ changes add too much complexity for a little gain.
GIL-free PyMem_Malloc() GIL-free PyMem_Malloc()
----------------------- -----------------------
When Python is compiled in debug mode, ``PyMem_Malloc()`` calls indirectly ``PyObject_Malloc()`` which requires the GIL to be held. When Python is compiled in debug mode, ``PyMem_Malloc()`` calls
indirectly ``PyObject_Malloc()`` which requires the GIL to be held.
That's why ``PyMem_Malloc()`` must be called with the GIL held. That's why ``PyMem_Malloc()`` must be called with the GIL held.
This PEP proposes to "fix" ``PyMem_Malloc()`` to make it always call This PEP proposes to "fix" ``PyMem_Malloc()`` to make it always call
@ -478,64 +489,65 @@ This PEP proposes to "fix" ``PyMem_Malloc()`` to make it always call
``PyMem_Malloc()``. ``PyMem_Malloc()``.
Allowing to call ``PyMem_Malloc()`` without holding the GIL might break Allowing to call ``PyMem_Malloc()`` without holding the GIL might break
applications which setup their own allocators or allocator hooks. Holding the applications which setup their own allocators or allocator hooks.
GIL is convinient to develop a custom allocator: no need to care of other Holding the GIL is convinient to develop a custom allocator: no need to
threads. It is also convinient for a debug allocator hook: Python internal care of other threads. It is also convinient for a debug allocator hook:
objects can be safetly inspected. Python internal objects can be safetly inspected.
Calling ``PyGILState_Ensure()`` in a memory allocator may have unexpected Calling ``PyGILState_Ensure()`` in a memory allocator may have
behaviour, especially at Python startup and at creation of a new Python thread unexpected behaviour, especially at Python startup and at creation of a
state. new Python thread state.
Don't add PyMem_RawMalloc() Don't add PyMem_RawMalloc()
--------------------------- ---------------------------
Replace ``malloc()`` with ``PyMem_Malloc()``, but only if the GIL is held. Replace ``malloc()`` with ``PyMem_Malloc()``, but only if the GIL is
Otherwise, keep ``malloc()`` unchanged. held. Otherwise, keep ``malloc()`` unchanged.
The ``PyMem_Malloc()`` is used without the GIL held in some Python functions. The ``PyMem_Malloc()`` is used without the GIL held in some Python
For example, the ``main()`` and ``Py_Main()`` functions of Python call functions. For example, the ``main()`` and ``Py_Main()`` functions of
``PyMem_Malloc()`` whereas the GIL do not exist yet. In this case, Python call ``PyMem_Malloc()`` whereas the GIL do not exist yet. In this
``PyMem_Malloc()`` should be replaced with ``malloc()`` (or case, ``PyMem_Malloc()`` should be replaced with ``malloc()`` (or
``PyMem_RawMalloc()``). ``PyMem_RawMalloc()``).
If an hook is used to the track memory usage, the ``malloc()`` memory will not If an hook is used to the track memory usage, the ``malloc()`` memory
be seen. Remaining ``malloc()`` may allocate a lot of memory and so would be will not be seen. Remaining ``malloc()`` may allocate a lot of memory
missed in reports. and so would be missed in reports.
Use existing debug tools to analyze the memory Use existing debug tools to analyze the memory
---------------------------------------------- ----------------------------------------------
There are many existing debug tools to analyze the memory. Some examples: There are many existing debug tools to analyze the memory. Some
`Valgrind <http://valgrind.org/>`_, examples: `Valgrind <http://valgrind.org/>`_, `Purify
`Purify <http://ibm.com/software/awdtools/purify/>`_, <http://ibm.com/software/awdtools/purify/>`_, `Clang AddressSanitizer
`Clang AddressSanitizer <http://code.google.com/p/address-sanitizer/>`_, <http://code.google.com/p/address-sanitizer/>`_, `failmalloc
`failmalloc <http://www.nongnu.org/failmalloc/>`_, <http://www.nongnu.org/failmalloc/>`_, etc.
etc.
The problem is to retrieve the Python object related to a memory pointer to read The problem is to retrieve the Python object related to a memory pointer
its type and/or content. Another issue is to retrieve the location of the to read its type and/or content. Another issue is to retrieve the
memory allocation: the C backtrace is usually useless (same reasoning than location of the memory allocation: the C backtrace is usually useless
macros using ``__FILE__`` and ``__LINE__``), the Python filename and line (same reasoning than macros using ``__FILE__`` and ``__LINE__``), the
number (or even the Python traceback) is more useful. Python filename and line number (or even the Python traceback) is more
useful.
Classic tools are unable to introspect Python internals to collect such Classic tools are unable to introspect Python internals to collect such
information. Being able to setup a hook on allocators called with the GIL held information. Being able to setup a hook on allocators called with the
allow to collect a lot of useful data from Python internals. GIL held allow to collect a lot of useful data from Python internals.
Add msize() Add msize()
----------- -----------
Add another field to ``PyMemBlockAllocator`` and ``PyMemMappingAllocator``:: Add another field to ``PyMemBlockAllocator`` and
``PyMemMappingAllocator``::
size_t msize(void *ptr); size_t msize(void *ptr);
This function returns the size of a memory block or a memory mapping. Return This function returns the size of a memory block or a memory mapping.
(size_t)-1 if the function is not implemented or if the pointer is unknown Return (size_t)-1 if the function is not implemented or if the pointer
(ex: NULL pointer). is unknown (ex: NULL pointer).
On Windows, this function can be implemented using ``_msize()`` and On Windows, this function can be implemented using ``_msize()`` and
``VirtualQuery()``. ``VirtualQuery()``.
@ -544,24 +556,25 @@ On Windows, this function can be implemented using ``_msize()`` and
No context argument No context argument
------------------- -------------------
Simplify the signature of allocator functions, remove the context argument: Simplify the signature of allocator functions, remove the context
argument:
* ``void* malloc(size_t size)`` * ``void* malloc(size_t size)``
* ``void* realloc(void *ptr, size_t new_size)`` * ``void* realloc(void *ptr, size_t new_size)``
* ``void free(void *ptr)`` * ``void free(void *ptr)``
It is likely for an allocator hook to be reused for ``PyMem_SetAllocator()`` It is likely for an allocator hook to be reused for
and ``PyObject_SetAllocator()``, or even ``PyMem_SetRawAllocator()``, but the ``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()``, or even
hook must call a different function depending on the allocator. The context is ``PyMem_SetRawAllocator()``, but the hook must call a different function
a convenient way to reuse the same custom allocator or hook for different depending on the allocator. The context is a convenient way to reuse the
Python allocators. same custom allocator or hook for different Python allocators.
External libraries External libraries
================== ==================
Python should try to reuse the same prototypes for allocator functions than Python should try to reuse the same prototypes for allocator functions
other libraries. than other libraries.
Libraries used by Python: Libraries used by Python:
@ -586,36 +599,41 @@ See also the `GNU libc: Memory Allocation Hooks
Memory allocators Memory allocators
================= =================
The C standard library provides the well known ``malloc()`` function. Its The C standard library provides the well known ``malloc()`` function.
implementation depends on the platform and of the C library. The GNU C library Its implementation depends on the platform and of the C library. The GNU
uses a modified ptmalloc2, based on "Doug Lea's Malloc" (dlmalloc). FreeBSD C library uses a modified ptmalloc2, based on "Doug Lea's Malloc"
uses `jemalloc <http://www.canonware.com/jemalloc/>`_. Google provides (dlmalloc). FreeBSD uses `jemalloc
tcmalloc which is part of `gperftools <http://code.google.com/p/gperftools/>`_. <http://www.canonware.com/jemalloc/>`_. Google provides tcmalloc which
is part of `gperftools <http://code.google.com/p/gperftools/>`_.
``malloc()`` uses two kinds of memory: heap and memory mappings. Memory ``malloc()`` uses two kinds of memory: heap and memory mappings. Memory
mappings are usually used for large allocations (ex: larger than 256 KB), mappings are usually used for large allocations (ex: larger than 256
whereas the heap is used for small allocations. KB), whereas the heap is used for small allocations.
On UNIX, the heap is handled by ``brk()`` and ``sbrk()`` system calls on Linux, On UNIX, the heap is handled by ``brk()`` and ``sbrk()`` system calls on
and it is contiguous. On Windows, the heap is handled by ``HeapAlloc()`` and Linux, and it is contiguous. On Windows, the heap is handled by
may be discontiguous. Memory mappings are handled by ``mmap()`` on UNIX and ``HeapAlloc()`` and may be discontiguous. Memory mappings are handled by
``VirtualAlloc()`` on Windows, they may be discontiguous. ``mmap()`` on UNIX and ``VirtualAlloc()`` on Windows, they may be
discontiguous.
Releasing a memory mapping gives back immediatly the memory to the system. On Releasing a memory mapping gives back immediatly the memory to the
UNIX, heap memory is only given back to the system if it is at the end of the system. On UNIX, heap memory is only given back to the system if it is
heap. Otherwise, the memory will only be given back to the system when all the at the end of the heap. Otherwise, the memory will only be given back to
memory located after the released memory are also released. the system when all the memory located after the released memory are
also released.
To allocate memory in the heap, the allocator tries to reuse free space. If To allocate memory in the heap, the allocator tries to reuse free space.
there is no contiguous space big enough, the heap must be increased, even if we If there is no contiguous space big enough, the heap must be increased,
have more free space than required size. This issue is called the "memory even if we have more free space than required size. This issue is
fragmentation": the memory usage seen by the system may be much higher than called the "memory fragmentation": the memory usage seen by the system
real usage. On Windows, ``HeapAlloc()`` creates a new memory mapping with may be much higher than real usage. On Windows, ``HeapAlloc()`` creates
``VirtualAlloc()`` if there is not enough free contiguous memory. a new memory mapping with ``VirtualAlloc()`` if there is not enough free
contiguous memory.
CPython has a *pymalloc* allocator for allocations smaller than 512 bytes. This CPython has a *pymalloc* allocator for allocations smaller than 512
allocator is optimized for small objects with a short lifetime. It uses memory bytes. This allocator is optimized for small objects with a short
mappings called "arenas" with a fixed size of 256 KB. lifetime. It uses memory mappings called "arenas" with a fixed size of
256 KB.
Other allocators: Other allocators:
@ -641,10 +659,10 @@ CPython issues related to memory allocation:
<http://bugs.python.org/issue13483>`_ <http://bugs.python.org/issue13483>`_
* `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which * `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which
isn't thread safe <http://bugs.python.org/issue16742>`_ isn't thread safe <http://bugs.python.org/issue16742>`_
* `Issue #18203: Replace calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc() * `Issue #18203: Replace calls to malloc() with PyMem_Malloc() or
<http://bugs.python.org/issue18203>`_ PyMem_RawMalloc() <http://bugs.python.org/issue18203>`_
* `Issue #18227: Use Python memory allocators in external libraries like zlib * `Issue #18227: Use Python memory allocators in external libraries like
or OpenSSL <http://bugs.python.org/issue18227>`_ zlib or OpenSSL <http://bugs.python.org/issue18227>`_
Projects analyzing the memory usage of Python applications: Projects analyzing the memory usage of Python applications: