PEP 445: cleanup

Avoid "should", "may" and "might". Rephrase some sentences
This commit is contained in:
Victor Stinner 2013-07-01 22:29:08 +02:00
parent feef345e1e
commit 03210e6ffd
1 changed files with 114 additions and 104 deletions

View File

@ -12,7 +12,8 @@ Python-Version: 3.4
Abstract
========
Add new APIs to customize Python memory allocators.
Add new Application Programming Interfaces (API) to customize Python
memory allocators.
Rationale
@ -20,21 +21,22 @@ Rationale
Use cases:
* Application embedding Python may want to isolate Python memory from
the memory of the application, or may want to use a different memory
* Applications embedding Python which want to isolate Python memory from
the memory of the application, or want to use a different memory
allocator optimized for its Python usage
* Python running on embedded devices with low memory and slow CPU.
A custom memory allocator may be required to use efficiently the
memory and/or to be able to use all the memory of the device.
* Debug tool to:
A custom memory allocator can be used for efficiency and/or to get
access all the memory of the device.
* Debug tools for memory allocators:
- track the memory usage (memory leaks)
- get the Python filename and line number where an object was
allocated
- detect buffer underflow, buffer overflow and detect misuse of Python
allocator APIs (builtin Python debug hooks)
- force allocation to fail to test handling of ``MemoryError``
exception
- track the memory usage (find memory leaks)
- get the location of a memory allocation: Python filename and line
number, and the size of a memory block
- detect buffer underflow, buffer overflow and misuse of Python
allocator APIs (see `Redesign Debug Checks on Memory Block
Allocators as Hooks`_)
- force memory allocations to fail to test handling of the
``MemoryError`` exception
Proposal
@ -56,8 +58,7 @@ New Functions and Structures
* Add a new ``PyMemAllocator`` structure::
typedef struct {
/* user context passed as the first argument
to the 3 functions */
/* user context passed as the first argument to the 3 functions */
void *ctx;
/* allocate a memory block */
@ -82,7 +83,7 @@ New Functions and Structures
- ``PYMEM_DOMAIN_OBJ``: ``PyObject_Malloc()``, ``PyObject_Realloc()``
and ``PyObject_Free()``
* Add new functions to get and set memory allocators:
* Add new functions to get and set memory block allocators:
- ``void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)``
- ``void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)``
@ -94,8 +95,7 @@ New Functions and Structures
* Add a new ``PyObjectArenaAllocator`` structure::
typedef struct {
/* user context passed as the first argument
to the 2 functions */
/* user context passed as the first argument to the 2 functions */
void *ctx;
/* allocate an arena */
@ -111,18 +111,17 @@ New Functions and Structures
- ``void PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)``
- ``void PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)``
* Add a new function to setup the debug checks on memory allocators when
a memory allocator is replaced:
* Add a new function to reinstall the debug checks on memory allocators when
a memory allocator is replaced with ``PyMem_SetAllocator()``:
- ``void PyMem_SetupDebugHooks(void)``
- Install the debug hook on all memory block allocators. The function
can be called more than once, hooks are not reinstalled if they
were already installed.
- The function does nothing is Python is not compiled in debug mode
- Install the debug hooks on all memory block allocators. The function can be
called more than once, hooks are only installed once.
- The function does nothing is Python is not compiled in debug mode.
* Memory allocators always returns *NULL* if size is greater than
``PY_SSIZE_T_MAX``. The check is done before calling the
inner function.
* Memory block allocators always return *NULL* if *size* is greater than
``PY_SSIZE_T_MAX``. The check is done before calling the inner
function.
The *pymalloc* allocator is optimized for objects smaller than 512 bytes
with a short lifetime. It uses memory mappings with a fixed size of 256
@ -140,8 +139,8 @@ Default allocators:
and ``free()``
Redesign Debug Checks on Memory Allocators as Hooks
----------------------------------------------------
Redesign Debug Checks on Memory Block Allocators as Hooks
---------------------------------------------------------
Since Python 2.3, Python implements different checks on memory
allocators in debug mode:
@ -157,7 +156,8 @@ In Python 3.3, the checks are installed by replacing ``PyMem_Malloc()``,
``PyMem_Realloc()``, ``PyMem_Free()``, ``PyObject_Malloc()``,
``PyObject_Realloc()`` and ``PyObject_Free()`` using macros. The new
allocator allocates a larger buffer and write a pattern to detect buffer
underflow and overflow. It uses the original ``PyObject_Malloc()``
underflow, buffer overflow and use after free (fill the buffer with the
pattern ``0xDB``). It uses the original ``PyObject_Malloc()``
function to allocate memory. So ``PyMem_Malloc()`` and
``PyMem_Realloc()`` call indirectly ``PyObject_Malloc()`` and
``PyObject_Realloc()``.
@ -178,13 +178,14 @@ Call traces when the hooks are installed (debug mode):
* ``PyObject_Free()`` => ``_PyMem_DebugFree()``
=> ``_PyObject_Free()``
As a result, ``PyMem_Malloc()`` and ``PyMem_Realloc()`` now always call
``malloc()`` and ``realloc()``, instead of calling ``PyObject_Malloc()``
and ``PyObject_Realloc()`` in debug mode.
As a result, ``PyMem_Malloc()`` and ``PyMem_Realloc()`` now call
``malloc()`` and ``realloc()`` in release mode and in debug mode,
instead of calling ``PyObject_Malloc()`` and ``PyObject_Realloc()`` in
debug mode.
When at least one memory allocator is replaced with
``PyMem_SetAllocator()``, the ``PyMem_SetupDebugHooks()`` function must
be called to install the debug hooks on top on the new allocator.
be called to reinstall the debug hooks on top on the new allocator.
Don't call malloc() directly anymore
@ -195,7 +196,7 @@ Don't call malloc() directly anymore
``PyObject_Realloc()`` falls back on ``PyMem_Realloc()`` instead of
``realloc()``
Replace direct calls to ``malloc()`` with ``PyMem_Malloc()``, or
Direct calls to ``malloc()`` are replaced with ``PyMem_Malloc()``, or
``PyMem_RawMalloc()`` if the GIL is not held.
Configure external libraries like zlib or OpenSSL to allocate memory
@ -205,22 +206,22 @@ Python is embedded in an application.
For the "track memory usage" use case, it is important to track memory
allocated in external libraries to have accurate reports, because these
allocations may be large.
allocations can be large (can raise a ``MemoryError`` exception).
If an hook is used to the track memory usage, the memory allocated by
``malloc()`` will not be tracked. Remaining ``malloc()`` in external
libraries like OpenSSL or bz2 may allocate large memory blocks and so
would be missed in memory usage reports.
direct calls to ``malloc()`` will not be tracked. Remaining ``malloc()``
in external libraries like OpenSSL or bz2 can allocate large memory
blocks and so would be missed in memory usage reports.
Examples
========
Use case 1: Replace Memory Allocator, keep pymalloc
Use case 1: Replace Memory Allocators, keep pymalloc
----------------------------------------------------
Dummy example wasting 2 bytes per memory block,
and 10 bytes per memory mapping::
and 10 bytes per *pymalloc* arena::
#include <stdlib.h>
@ -267,6 +268,7 @@ and 10 bytes per memory mapping::
PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc);
PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &alloc);
/* leave PYMEM_DOMAIN_OBJ unchanged, use pymalloc */
arena.ctx = &arena_padding;
arena.alloc = my_alloc_arena;
@ -277,7 +279,7 @@ and 10 bytes per memory mapping::
}
Use case 2: Replace Memory Allocator, override pymalloc
Use case 2: Replace Memory Allocators, override pymalloc
--------------------------------------------------------
If your allocator is optimized for allocations of objects smaller than
@ -322,11 +324,14 @@ Dummy example wasting 2 bytes per memory block::
PyMem_SetupDebugHooks();
}
The *pymalloc* arena does not need to be replaced, because it is no more
used by the new allocator.
Use case 3: Setup Allocator Hooks
---------------------------------
Example to setup hooks on all memory allocators::
Use case 3: Setup Hooks On Memory Block Allocators
--------------------------------------------------
Example to setup hooks on all memory block allocators::
struct {
PyMemAllocator raw;
@ -390,22 +395,23 @@ Example to setup hooks on all memory allocators::
}
.. note::
``PyMem_SetupDebugHooks()`` does not need to be called because the
allocator is not replaced: Python debug hooks are installed
automatically at startup.
``PyMem_SetupDebugHooks()`` does not need to be called because
memory allocator are not replaced: the debug checks on memory
block allocators are installed automatically at startup.
Performances
============
The implementation of this PEP (issue #3329) has no visible overhead on
the Python benchmark suite.
Results of the `Python benchmarks suite
<http://hg.python.org/benchmarks>`_ (-b 2n3): some tests are 1.04x
faster, some tests are 1.04 slower, significant is between 115 and -191.
faster, some tests are 1.04 slower. Results of pybench microbenchmark:
"+0.1%" slower globally (diff between -4.9% and +5.6%).
Results of pybench benchmark: "+0.1%" slower globally (diff between
-4.9% and +5.6%).
The full reports are attached to the issue #3329.
The full output of benchmarks is attached to the issue #3329.
Rejected Alternatives
@ -428,8 +434,9 @@ with:
* ``void PyMem_SetAllocator(PyMemAllocator *allocator)``
* ``void PyObject_SetAllocator(PyMemAllocator *allocator)``
With more specific functions, it becomes more difficult to write generic
code, like reusing the same code for different allocator domains.
This alternative was rejected because it is not possible to write
generic code with more specific functions: code must be duplicated for
each memory allocator domain.
Make PyMem_Malloc() reuse PyMem_RawMalloc() by default
@ -439,25 +446,25 @@ If ``PyMem_Malloc()`` would call ``PyMem_RawMalloc()`` by default,
calling ``PyMem_SetAllocator(PYMEM_DOMAIN_RAW, alloc)`` would also also
patch ``PyMem_Malloc()`` indirectly.
This option was rejected because ``PyMem_SetAllocator()`` would have a
different behaviour depending on the domain. Always having the same
behaviour is less error-prone.
This alternative was rejected because ``PyMem_SetAllocator()`` would
have a different behaviour depending on the domain. Always having the
same behaviour is less error-prone.
Add a new PYDEBUGMALLOC environment variable
--------------------------------------------
To be able to use the Python builtin debug hooks even when a custom
memory allocator replaces the default Python allocator, an environment
variable ``PYDEBUGMALLOC`` can be added to setup these debug function
hooks, instead of adding the new function ``PyMem_SetupDebugHooks()``.
If the environment variable is present, ``PyMem_SetRawAllocator()``,
``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()`` will reinstall
automatically the hook on top of the new allocator.
Add a new ``PYDEBUGMALLOC`` environment variable to enable debug checks
on memory block allocators. The environment variable replaces the new
function ``PyMem_SetupDebugHooks()`` which is not needed anymore.
Another advantage is to allow to enable debug checks even in release
mode: debug checks are always compiled, but only enabled when the
environment variable is present and non-empty.
A new environment variable would make the Python initialization even
more complex. The `PEP 432 <http://www.python.org/dev/peps/pep-0432/>`_
tries to simply the CPython startup sequence.
This alternative was rejected because a new environment variable would
make the Python initialization even more complex. The `PEP 432
<http://www.python.org/dev/peps/pep-0432/>`_ tries to simply the CPython
startup sequence.
Use macros to get customizable allocators
@ -503,7 +510,7 @@ Example of ``PyMem_Malloc`` macro with the modified
void* _PyMem_MallocTrace(const char *filename, int lineno,
size_t size);
/* need also a function for the Python stable ABI */
/* the function is still needed for the Python stable ABI */
void* PyMem_Malloc(size_t size);
#define PyMem_Malloc(size) \
@ -527,19 +534,19 @@ In Python 3.3, when Python is compiled in debug mode, ``PyMem_Malloc()``
calls indirectly ``PyObject_Malloc()`` which requires the GIL to be
held. That's why ``PyMem_Malloc()`` must be called with the GIL held.
This PEP proposes changes ``PyMem_Malloc()``: it now always call
``malloc()``. The "GIL must be held" restriction can be removed from
This PEP changes ``PyMem_Malloc()``: it now always call ``malloc()``.
The "GIL must be held" restriction could be removed from
``PyMem_Malloc()``.
This alternative was rejected because allowing to call
``PyMem_Malloc()`` without holding the GIL might break applications
``PyMem_Malloc()`` without holding the GIL can break applications
which setup their own allocators or allocator hooks. Holding the GIL is
convinient to develop a custom allocator: no need to care of other
threads. It is also convinient for a debug allocator hook: Python
internal objects can be safetly inspected.
Calling ``PyGILState_Ensure()`` in
a memory allocator may have unexpected behaviour, especially at Python
a memory allocator has unexpected behaviour, especially at Python
startup and at creation of a new Python thread state.
@ -552,13 +559,14 @@ held. Otherwise, keep ``malloc()`` unchanged.
The ``PyMem_Malloc()`` is used without the GIL held in some Python
functions. For example, the ``main()`` and ``Py_Main()`` functions of
Python call ``PyMem_Malloc()`` whereas the GIL do not exist yet. In this
case, ``PyMem_Malloc()`` should be replaced with ``malloc()`` (or
case, ``PyMem_Malloc()`` would be replaced with ``malloc()`` (or
``PyMem_RawMalloc()``).
If an hook is used to the track memory usage, the memory allocated by
direct calls to ``malloc()`` will not be tracked. External libraries
like OpenSSL or bz2 should not call ``malloc()`` directly, so large
allocated will be included in memory usage reports.
This alternative was rejected because ``PyMem_RawMalloc()`` is required
for accurate reports of the memory usage. When a debug hook is used to
track the memory usage, the memory allocated by direct calls to
``malloc()`` cannot be tracked. ``PyMem_RawMalloc()`` can be hooked and
so all the memory allocated by Python can be tracked.
Use existing debug tools to analyze the memory
@ -571,11 +579,11 @@ examples: `Valgrind <http://valgrind.org/>`_, `Purify
<http://www.nongnu.org/failmalloc/>`_, etc.
The problem is to retrieve the Python object related to a memory pointer
to read its type and/or content. Another issue is to retrieve the
to read its type and/or its content. Another issue is to retrieve the
location of the memory allocation: the C backtrace is usually useless
(same reasoning than macros using ``__FILE__`` and ``__LINE__``), the
Python filename and line number (or even the Python traceback) is more
useful.
(same reasoning than macros using ``__FILE__`` and ``__LINE__``, see
`Pass the C filename and line number`_), the Python filename and line
number (or even the Python traceback) is more useful.
This alternative was rejected because classic tools are unable to
introspect Python internals to collect such information. Being able to
@ -586,8 +594,8 @@ lot of useful data from Python internals.
Add a msize() function
----------------------
Add another field to ``PyMemAllocator`` and ``PyObjectArenaAllocator``
structures::
Add another function to ``PyMemAllocator`` and
``PyObjectArenaAllocator`` structures::
size_t msize(void *ptr);
@ -607,8 +615,8 @@ The additional ``msize()`` function was rejected because only few
platforms implement it. For example, Linux with the GNU libc does not
provide a function to get the size of a memory block. ``msize()`` is not
currently used in the Python source code. The function is only used to
track the memory usage, but makes the API more complex. A debug hook can
implemente the function internally, there is no need to add it to
track the memory usage, and makes the API more complex. A debug hook can
implement the function internally, there is no need to add it to
``PyMemAllocator`` and ``PyObjectArenaAllocator`` structures.
@ -653,17 +661,19 @@ Libraries used by Python:
* lzma: `LZMA SDK - How to Use
<http://www.asawicki.info/news_1368_lzma_sdk_-_how_to_use.html>`_,
pass an opaque pointer
* lipmpdec doesn't have this extra *ctx* parameter
* lipmpdec: no opaque pointer (classic malloc API)
Other libraries:
* glib: `g_mem_set_vtable()
<http://developer.gnome.org/glib/unstable/glib-Memory-Allocation.html#g-mem-set-vtable>`_
* libxml2: `xmlGcMemSetup() <http://xmlsoft.org/html/libxml-xmlmemory.html>`_,
* libxml2:
`xmlGcMemSetup() <http://xmlsoft.org/html/libxml-xmlmemory.html>`_,
global
* Oracle's OCI: `Oracle Call Interface Programmer's Guide,
Release 2 (9.2)
<http://docs.oracle.com/cd/B10501_01/appdev.920/a96584/oci15re4.htm>`_
<http://docs.oracle.com/cd/B10501_01/appdev.920/a96584/oci15re4.htm>`_,
pass an opaque pointer
See also the `GNU libc: Memory Allocation Hooks
<http://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html>`_.
@ -676,30 +686,30 @@ The C standard library provides the well known ``malloc()`` function.
Its implementation depends on the platform and of the C library. The GNU
C library uses a modified ptmalloc2, based on "Doug Lea's Malloc"
(dlmalloc). FreeBSD uses `jemalloc
<http://www.canonware.com/jemalloc/>`_. Google provides tcmalloc which
<http://www.canonware.com/jemalloc/>`_. Google provides *tcmalloc* which
is part of `gperftools <http://code.google.com/p/gperftools/>`_.
``malloc()`` uses two kinds of memory: heap and memory mappings. Memory
mappings are usually used for large allocations (ex: larger than 256
KB), whereas the heap is used for small allocations.
On UNIX, the heap is handled by ``brk()`` and ``sbrk()`` system calls on
Linux, and it is contiguous. On Windows, the heap is handled by
``HeapAlloc()`` and may be discontiguous. Memory mappings are handled by
``mmap()`` on UNIX and ``VirtualAlloc()`` on Windows, they may be
On UNIX, the heap is handled by ``brk()`` and ``sbrk()`` system calls,
and it is contiguous. On Windows, the heap is handled by
``HeapAlloc()`` and can be discontiguous. Memory mappings are handled by
``mmap()`` on UNIX and ``VirtualAlloc()`` on Windows, they can be
discontiguous.
Releasing a memory mapping gives back immediatly the memory to the
system. On UNIX, heap memory is only given back to the system if it is
at the end of the heap. Otherwise, the memory will only be given back to
the system when all the memory located after the released memory are
also released.
system. On UNIX, the heap memory is only given back to the system if the
released block is located at the end of the heap. Otherwise, the memory
will only be given back to the system when all the memory located after
the released memory is also released.
To allocate memory in the heap, the allocator tries to reuse free space.
If there is no contiguous space big enough, the heap must be increased,
even if we have more free space than required size. This issue is
To allocate memory on the heap, an allocator tries to reuse free space.
If there is no contiguous space big enough, the heap must be enlarged,
even if there is more free space than required size. This issue is
called the "memory fragmentation": the memory usage seen by the system
may be much higher than real usage. On Windows, ``HeapAlloc()`` creates
is higher than real usage. On Windows, ``HeapAlloc()`` creates
a new memory mapping with ``VirtualAlloc()`` if there is not enough free
contiguous memory.
@ -730,8 +740,8 @@ CPython issues related to memory allocation:
<http://bugs.python.org/issue3329>`_
* `Issue #13483: Use VirtualAlloc to allocate memory arenas
<http://bugs.python.org/issue13483>`_
* `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which
isn't thread safe <http://bugs.python.org/issue16742>`_
* `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline,
which isn't thread safe <http://bugs.python.org/issue16742>`_
* `Issue #18203: Replace calls to malloc() with PyMem_Malloc() or
PyMem_RawMalloc() <http://bugs.python.org/issue18203>`_
* `Issue #18227: Use Python memory allocators in external libraries like