python-peps/pep-0454.txt

PEP: 454
Title: Add a new tracemalloc module to trace Python memory allocations
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 3-September-2013
Python-Version: 3.4


Abstract
========

This PEP proposes to add a new ``tracemalloc`` module to trace memory
blocks allocated by Python.


Rationale
=========

Classic generic tools like Valgrind can get the C traceback where a
memory block was allocated. Using such tools to analyze Python memory
allocations does not help because most memory blocks are allocated in
the same C function, in ``PyMem_Malloc()`` for example. Moreover, Python
has an allocator for small objects called "pymalloc" which keeps free
blocks for efficiency. This is not well handled by these tools.

There are debug tools dedicated to the Python language like ``Heapy``
``Pympler`` and ``Meliae`` which lists all alive objects using the
garbage collector module (functions like ``gc.get_objects()``,
``gc.get_referrers()`` and ``gc.get_referents()``), compute their size
(ex: using ``sys.getsizeof()``) and group objects by type. These tools
provide a better estimation of the memory usage of an application.  They
are useful when most memory leaks are instances of the same type and
this type is only instantiated in a few functions. Problems arise when
the object type is very common like ``str`` or ``tuple``, and it is hard
to identify where these objects are instantiated.

Finding reference cycles is also a difficult problem.  There are
different tools to draw a diagram of all references.  These tools
cannot be used on large applications with thousands of objects because
the diagram is too huge to be analyzed manually.


Proposal
========

Using the customized allocation API from PEP 445, it becomes easy to
set up a hook on Python memory allocators. A hook can inspect Python
internals to retrieve Python tracebacks. The idea of getting the current
traceback comes from the faulthandler module. The faulthandler dumps
the traceback of all Python threads on a crash, here is the idea is to
get the traceback of the current Python thread when a memory block is
allocated by Python.

This PEP proposes to add a new ``tracemalloc`` module, a debug tool
to trace memory blocks allocated by Python. The module provides the
following information:

* Traceback where an object was allocated
* Statistics on allocated memory blocks per filename and per line
  number: total size, number and average size of allocated memory blocks
* Computed differences between two snapshots to detect memory leaks

The API of the tracemalloc module is similar to the API of the faulthandler
module: ``enable()`` / ``start()``, ``disable()`` / ``stop()`` and
``is_enabled()`` / ``is_tracing()`` functions, an environment variable
(``PYTHONFAULTHANDLER`` and ``PYTHONTRACEMALLOC``), and a ``-X`` command line
option (``-X faulthandler`` and ``-X tracemalloc``). See the `documentation of
the faulthandler module <http://docs.python.org/3/library/faulthandler.html>`_.

The idea of tracing memory allocations is not new. It was first
implemented in the PySizer project in 2005. PySizer was implemented
differently: the traceback was stored in frame objects and some Python
types were linked the trace with the name of object type. PySizer patch
on CPython adds a overhead on performances and memory footprint, even if
the PySizer was not used. tracemalloc attachs a traceback to the
underlying layer, to memory blocks, and has no overhead when the module
is not tracing memory allocations.

The tracemalloc module has been written for CPython. Other
implementations of Python may not be able to provide it.


API
===

To trace most memory blocks allocated by Python, the module should be
started as early as possible by setting the ``PYTHONTRACEMALLOC``
environment variable to ``1``, or by using ``-X tracemalloc`` command
line option. The ``tracemalloc.start()`` function can be called at
runtime to start tracing Python memory allocations.

By default, a trace of an allocated memory block only stores the most
recent frame (1 frame). To store 25 frames at startup: set the
``PYTHONTRACEMALLOC`` environment variable to ``25``, or use the ``-X
tracemalloc=25`` command line option. The ``set_traceback_limit()``
function can be used at runtime to set the limit.

By default, Python memory blocks allocated in the ``tracemalloc`` module
are ignored using a filter. Use ``clear_filters()`` to trace also these
memory allocations.


Main functions
--------------

``clear_traces()`` function:

    Clear traces of memory blocks allocated by Python.

    See also ``stop()``.


``get_traced_memory()`` function:

    Get the current size and maximum size of memory blocks traced by the
    ``tracemalloc`` module as a tuple: ``(size: int, max_size: int)``.


``get_tracemalloc_memory()`` function:

    Get the memory usage in bytes of the ``tracemalloc`` module used to
    store traces of memory blocks. Return an ``int``.


``is_tracing()`` function:

    ``True`` if the ``tracemalloc`` module is tracing Python memory
    allocations, ``False`` otherwise.

    See also ``start()`` and ``stop()`` functions.


``start()`` function:

    Start tracing Python memory allocations.

    The function installs hooks on Python memory allocators. These hooks
    have important overhead in term of performances and memory usage:
    see `Filter functions`_ to limit the overhead.

    See also ``stop()`` and ``is_tracing()`` functions.


``stop()`` function:

    Stop tracing Python memory allocations and clear traces of memory
    blocks allocated by Python.

    The function uninstalls hooks on Python memory allocators, so the
    overhead of the module becomes null.

    Call ``get_traces()`` or ``take_snapshot()`` function to get traces
    before clearing them.

    See also ``start()`` and ``is_tracing()`` functions.


``take_snapshot()`` function:

    Take a snapshot of traces of memory blocks allocated by Python using
    the ``get_traces()`` function. Return a new ``Snapshot`` instance.

    The ``tracemalloc`` module must be tracing memory allocations to
    take a snapshot, see the the ``start()`` function.

    See also ``get_traces()`` and ``get_object_traceback()`` functions.


Trace functions
---------------

When Python allocates a memory block, ``tracemalloc`` attachs a "trace" to
the memory block to store its size in bytes and the traceback where the
allocation occurred.

The following functions give access to these traces. A trace is a ``(size: int,
traceback)`` tuple. *size* is the size of the memory block in bytes.
*traceback* is a tuple of frames sorted from the most recent to the oldest
frame, limited to ``get_traceback_limit()`` frames. A frame is
a ``(filename: str, lineno: int)`` tuple.

A traceback contains at least ``1`` frame. If the ``tracemalloc`` module
failed to get a frame, the ``"<unknown>"`` filename and the line number ``0``
are used. If it failed to get the traceback or if the traceback limit is ``0``,
the traceback is ``(('<unknown>', 0),)``.

Example of a trace: ``(32, (('x.py', 7), ('x.py', 11)))``.  The memory block
has a size of 32 bytes and was allocated at ``x.py:7``, line called from line
``x.py:11``.


``get_object_traceback(obj)`` function:

    Get the traceback where the Python object *obj* was allocated.
    Return a tuple of ``(filename: str, lineno: int)`` tuples.

    Return ``None`` if the ``tracemalloc`` module is not tracing memory
    allocations or did not trace the allocation of the object.

    See also ``gc.get_referrers()`` and ``sys.getsizeof()`` functions.


``get_traceback_limit()`` function:

    Get the maximum number of frames stored in the traceback of a trace.

    By default, a trace of a memory block only stores the most recent
    frame: the limit is ``1``.

    Use the ``set_traceback_limit()`` function to change the limit.


``get_traces()`` function:

    Get traces of memory blocks allocated by Python. Return a list of
    ``(size: int, traceback: tuple)`` tuples. *traceback* is a tuple of
    ``(filename: str, lineno: int)`` tuples.

    The list of traces does not include memory blocks allocated before
    the ``tracemalloc`` module started to trace memory allocations nor
    memory blocks ignored by filters (see ``get_filters()``).

    The list has an undefined order. Take a snapshot using
    ``take_snapshot()`` and use the ``Snapshot.statistics()`` method to
    get a sorted list of statistics.

    Tracebacks of traces are limited to ``traceback_limit`` frames. Use
    ``set_traceback_limit()`` to store more frames.

    Return an empty list if the ``tracemalloc`` module is not tracing
    memory allocations.

    See also ``take_snapshot()`` and ``get_object_traceback()``
    functions.


``set_traceback_limit(nframe: int)`` function:

    Set the maximum number of frames stored in the traceback of a trace.

    Storing the traceback of each memory allocation has an important
    overhead on the memory usage. Use the ``get_tracemalloc_memory()``
    function to measure the overhead and the ``add_filter()`` function
    to select which memory allocations are traced.

    If the limit is set to ``0`` frame, the traceback ``(('<unknown>',
    0),)`` will be used for all traces.

    Use the ``get_traceback_limit()`` function to get the current limit.

    The ``PYTHONTRACEMALLOC`` environment variable and the ``-X``
    ``tracemalloc=NFRAME`` command line option can be used to set a
    limit at startup.


Filter functions
----------------

Tracing all Python memroy allocations has an important overhead on performances
and on the memory usage.

To limit the overhead, some files can be excluded or tracing can be restricted
to a set of files using filters. Examples: ``add_filter(Filter(True,
subprocess.__file__))`` only traces memory allocations in the ``subprocess``
module, and ``add_filter(Filter(False, tracemalloc.__file__))`` ignores
memory allocations in the ``tracemalloc`` module

By default, there is one exclusive filter to ignore Python memory blocks
allocated by the ``tracemalloc`` module.

Use the ``get_tracemalloc_memory()`` function to measure the memory usage.
See also the ``set_traceback_limit()`` function to configure how many
frames are stored.

``add_filter(filter)`` function:

    Add a new filter on Python memory allocations, *filter* is a
    ``Filter`` instance.

    All inclusive filters are applied at once, a memory allocation is
    ignored if no inclusive filters match its trace. A memory allocation
    is ignored if at least one exclusive filter matchs its trace.

    The new filter is not applied on already collected traces. Use the
    ``clear_traces()`` function to ensure that all traces match the new
    filter.


``clear_filters()`` function:

    Clear the filter list.

    See also the ``get_filters()`` function.


``get_filters()`` function:

    Get the filters on Python memory allocations. Return a list of
    ``Filter`` instances.

    See also the ``clear_filters()`` function.


Filter
------

``Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False)`` class:

    Filter to select which memory allocations are traced. Filters can be
    used to reduce the memory usage of the ``tracemalloc`` module, which
    can be read using the ``get_tracemalloc_memory()`` function.

    The ``'*'`` joker character can be used in *filename_pattern* to
    match any substring, including empty string. The ``'.pyc'`` and
    ``'.pyo'`` file extensions are replaced with ``'.py'``. On Windows,
    the comparison is case insensitive and the alternative separator
    ``'/'`` is replaced with the standard separator ``'\'``.

    Use ``Filter(False, "<unknown>")`` to exclude empty tracebacks.

``inclusive`` attribute:

    If *inclusive* is ``True`` (include), only trace memory blocks
    allocated in a file with a name matching ``filename_pattern`` at
    line number ``lineno``.

    If *inclusive* is ``False`` (exclude), ignore memory blocks
    allocated in a file with a name matching ``filename_pattern`` at
    line number ``lineno``.

``lineno`` attribute:

    Line number (``int``) of the filter. If *lineno* is ``None``, the
    filter matches any line number.

``filename_pattern`` attribute:

    Filename pattern (``str``) of the filter.

``all_frames`` attribute:

    If *all_frames* is ``True``, all frames of the traceback are
    checked. If *all_frames* is ``False``, only the most recent frame is
    checked.

    This attribute is ignored if the traceback limit is less than ``2``.
    See the ``get_traceback_limit()`` function.


Snapshot
--------

``Snapshot(timestamp: datetime.datetime, traceback_limit: int, traces: dict=None)`` class:

    Snapshot of traces of memory blocks allocated by Python.

    The ``take_snapshot()`` function create a snapshot instance.

``apply_filters(filters)`` method:

    Apply filters on the ``traces`` dictionary, *filters* is a list of
    ``Filter`` instances. Return a new ``Snapshot`` instance with the
    filtered traces.

    If *filters* is an empty list, return a new ``Snapshot`` instance
    with a copy of the traces.


``dump(filename)`` method:

    Write the snapshot into a file.

    Use ``load()`` to reload the snapshot.


``load(filename)`` classmethod:

    Load a snapshot from a file.

    See also ``dump()``.


``statistics(group_by: str, cumulative: bool=False, compare_to=None)`` method:

    Get statistics as a sorted list of ``Statistic`` instances, grouped
    by *group_by*:

    =====================  ========================
    group_by               description
    =====================  ========================
    ``'filename'``         filename
    ``'lineno'``           filename and line number
    ``'traceback'``        traceback
    =====================  ========================

    If *cumulative* is ``True``, cumulate size and count of memory
    blocks of all frames of the traceback of a trace, not only the most
    recent frame. The cumulative mode can only be used with key types
    ``'filename'`` and ``'lineno'`` with ``traceback_limit`` greater
    than ``1``.

    If *compare_to* is set to a previous ``Snapshot`` instance, compute
    the differences betwen the two snapshots. Otherwise,
    ``Statistic.size_diff`` and ``Statistic.count_diff`` attributes are
    set to zero.

    The result is sorted from the biggest to the smallest by: absolute
    value of ``Statistic.size_diff``, ``Statistic.size``, absolute value
    of ``Statistic.count_diff``, ``Statistic.count`` and then by
    ``Statistic.key``.


``traceback_limit`` attribute:

    Maximum number of frames stored in the traceback of ``traces``: see
    the ``get_traceback_limit()`` function.

``traces`` attribute:

    Traces of all memory blocks allocated by Python: see the
    ``get_traces()`` function.

``timestamp`` attribute:

    Creation date and time of the snapshot, ``datetime.datetime``
    instance.


Statistic
---------

``Statistic(key, size, size_diff, count, count_diff)`` class:

    Statistic on memory allocations.

    ``size_diff`` and ``count_diff`` attributes are the difference
    between two ``Snapshot`` instance.

    ``Snapshot.statistics()`` returns a list of ``Statistic`` instances.

``traceback`` attribute:

    Tuple of ``(filename: str, lineno: int)`` tuples.

``count`` attribute:

    Number of memory blocks (``int``).

``count_diff`` attribute:

    Difference of number of memory blocks (``int``).

``size`` attribute:

    Total size of memory blocks in bytes (``int``).

``size_diff`` attribute:

    Difference of total size of memory blocks in bytes (``int``).


Rejected Alternatives
=====================

Log calls to the memory allocator
---------------------------------

A different approach is to log calls to ``malloc()``, ``realloc()`` and
``free()`` functions. Calls can be logged into a file or send to another
computer through the network. Example of a log entry: name of the
function, size of the memory block, address of the memory block, Python
traceback where the allocation occurred, timestamp.

Logs cannot be used directly, getting the current status of the memory
requires to parse previous logs. For example, it is not possible to get
directly the traceback of a Python object, like
``get_object_traceback(obj)`` does with traces.

Python uses objects with a very short lifetime and so makes an extensive
use of memory allocators. It has an allocator optimized for small
objects (less than 512 bytes) with a short lifetime.  For example, the
Python test suites calls ``malloc()``, ``realloc()`` or ``free()``
270,000 times per second in average. If the size of log entry is 32
bytes, logging produces 8.2 MB per second or 29.0 GB per hour.

The alternative was rejected because it is less efficient and has less
features. Parsing logs in a different process or a different computer is
slower than maintaining traces on allocated memory blocks in the same
process.


Prior Work
==========

* `Python Memory Validator
  <http://www.softwareverify.com/python/memory/index.html>`_ (2005-2013):
  commercial Python memory validator developed by Software Verification.
  It uses the Python Reflection API.
* `PySizer <http://pysizer.8325.org/>`_: Google Summer of Code 2005 project by
  Nick Smallbone.
* `Heapy
  <http://guppy-pe.sourceforge.net/>`_ (2006-2013):
  part of the Guppy-PE project written by Sverker Nilsson.
* Draft PEP: `Support Tracking Low-Level Memory Usage in CPython
  <http://svn.python.org/projects/python/branches/bcannon-sandboxing/PEP.txt>`_
  (Brett Canon, 2006)
* Muppy: project developed in 2008 by Robert Schuppenies.
* `asizeof <http://code.activestate.com/recipes/546530/>`_:
  a pure Python module to estimate the size of objects by Jean
  Brouwers (2008).
* `Heapmonitor <http://www.scons.org/wiki/LudwigHaehne/HeapMonitor>`_:
  It provides facilities to size individual objects and can track all objects
  of certain classes. It was developed in 2008 by Ludwig Haehne.
* `Pympler <http://code.google.com/p/pympler/>`_ (2008-2011):
  project based on asizeof, muppy and HeapMonitor
* `objgraph <http://mg.pov.lt/objgraph/>`_ (2008-2012)
* `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version
  of the CherryPy memory leak debugger, written by Marius Gedminas (2008-2013)
* `Meliae
  <https://pypi.python.org/pypi/meliae>`_:
  Python Memory Usage Analyzer developed by John A Meinel since 2009
* `gdb-heap <https://fedorahosted.org/gdb-heap/>`_: gdb script written in
  Python by Dave Malcom (2010-2011) to analyze the usage of the heap memory
* `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_:
  written by Fabian Pedregosa (2011-2013)
* `caulk <https://github.com/smartfile/caulk/>`_: written by Ben Timby in 2012

See also `Pympler Related Work
<http://pythonhosted.org/Pympler/related.html>`_.


Links
=====

tracemalloc:

* `#18874: Add a new tracemalloc module to trace Python
  memory allocations <http://bugs.python.org/issue18874>`_
* `pytracemalloc on PyPI
  <https://pypi.python.org/pypi/pytracemalloc>`_


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: