python-peps/pep-0454.txt

PEP: 454
Title: Add a new tracemalloc module to trace Python memory allocations
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 3-September-2013
Python-Version: 3.4


Abstract
========

Add a new ``tracemalloc`` module to trace Python memory allocations.



Rationale
=========

This PEP proposes to a new ``tracemalloc`` module, a debug tool to trace
memory allocations made by Python. The module provides the following
information:

* Statistics on allocations per Python line number (file and line):
  size, number, and average size of allocations
* Compute delta between two "snapshots"
* Location of a memory allocation: Python filename and line number

The ``tracemalloc`` module is different than other third-party modules
like ``Heapy`` or ``PySizer``, because it is focused on the location of
a memory allocation rather that the object type or object content.


API
===

To trace the most Python memory allocations, the module should be
enabled as early as possible in your application by calling
``tracemalloc.enable()`` function, by setting the ``PYTHONTRACEMALLOC``
environment variable to ``1``, or  by using ``-X tracemalloc`` command
line option.


Functions
---------

enable():

    Start tracing Python memory allocations.

disable():

    Stop tracing Python memory allocations and stop the timer started by
    ``start_timer()``.

is_enabled():

    Get the status of the module: ``True`` if it is enabled, ``False``
    otherwise.

get_object_trace(obj):

    Get the trace of a Python object *obj* as a namedtuple with 3 attributes:

    - ``size``: size in bytes of the object
    - ``filename``: name of the Python script where the object was allocated
    - ``lineno``: line number where the object was allocated

    Return ``None`` if tracemalloc did not save the location where the object
    was allocated, for example if tracemalloc was disabled.

get_process_memory():

    Get the memory usage of the current process as a meminfo namedtuple
    with two attributes:

    * ``rss``: Resident Set Size in bytes
    * ``vms``: size of the virtual memory in bytes

    Return ``None`` if the platform is not supported.

    Use the ``psutil`` module if available.

start_timer(delay: int, func: callable, args: tuple=(), kwargs: dict={}):

    Start a timer calling ``func(*args, **kwargs)`` every *delay*
    seconds.

    The timer is based on the Python memory allocator, it is not real
    time.  *func* is called after at least *delay* seconds, it is not
    called exactly after *delay* seconds if no Python memory allocation
    occurred.

    If ``start_timer()`` is called twice, previous parameters are
    replaced.  The timer has a resolution of 1 second.

    `start_timer()`` is used by ``DisplayTop`` and ``TakeSnapshot`` to
    run regulary a task.

stop_timer():

    Stop the timer started by ``start_timer()``.


DisplayTop class
----------------

``DisplayTop(count: int, file=sys.stdout)`` class:

    Display the list of the N biggest memory allocations.

``display()`` method:

    Display the top

``start(delay: int)`` method:

    Start a task using tracemalloc timer to display the top every delay seconds.

``stop()`` method:

    Stop the task started by the ``DisplayTop.start()`` method

``color`` attribute:

    ``display()`` uses colors if ``True`` (bool,
    default: ``stream.isatty()``).

``compare_with_previous`` attribute:

    If ``True``, ``display()`` compares with the previous top if
    ``True``. If ``False``, compare with the first one (bool, default:
    ``True``): .

``filename_parts`` attribute:

      Number of displayed filename parts (int, default: ``3``).

``show_average`` attribute:

      If ``True``, ``display()`` shows the average size of allocations
      (bool, default: ``True``).

``show_count`` attribute:

    If ``True``, ``display()`` shows the number of allocations (bool, default: ``True``).

``show_lineno`` attribute:

    If ``True``, use also the line number, not only the filename (bool,
    default: ``True``). If ``False``, only show the filename.

``show_size`` attribute:

    If ``True``, ``display()`` shows the size of allocations (bool,
    default: ``True``).

``user_data_callback`` attribute:

    Optional callback collecting user data (callable, default:
    ``None``).  See ``Snapshot.create()``.


Snapshot class
--------------

``Snapshot()`` class:

    Snapshot of Python memory allocations. Use ``TakeSnapshot`` to
    regulary take snapshots.

``create(user_data_callback=None)`` method:

    Take a snapshot. If user_data_callback is specified, it must be a
    callable object returning a list of (title: str, format: str, value:
    int). format must be "size". The list must always have the same
    length and the same order to be able to compute differences between
    values.

    Example: ``[('Video memory', 'size', 234902)]``.

``filter_filenames(patterns: str|list, include: bool)`` method:

    Remove filenames not matching any pattern if include is True, or
    remove filenames matching a pattern if include is False (exclude).
    See fnmatch.fnmatch() for the syntax of patterns.

``write(filename)`` method:

    Write the snapshot into a file.

``load(filename)`` classmethod:

    Load a snapshot from a file.

``process_memory`` attribute:

    Memory usage of the process, result of ``get_process_memory()``.
    It can be ``None``.

``user_data`` attribute:

    Optional list of user data, result of *user_data_callback* in
    ``Snapshot.create()`` (default: None).

``pid`` attribute:

    Identifier of the process which created the snapshot (int).

``stats`` attribute:

    Raw memory allocation statistics (dict).

``timestamp`` attribute:

    Date and time of the creation of the snapshot (str).


TakeSnapshot class
------------------

``TakeSnapshot`` class:

    Task taking snapshots of Python memory allocations: write them into files.

``start(delay: int)`` method:

    Start a task taking a snapshot every delay seconds.

``stop()`` method:

    Stop the task started by the ``TakeSnapshot.start()`` method.

``take_snapshot()`` method:

    Take a snapshot.

``filename_template`` attribute:

    Template (str) to create a filename. "Variables" can be used in the
    template:

    * ``$pid``: identifier of the current process
    * ``$timestamp``: current date and time
    * ``$counter``: counter starting at 1 and incremented at each snapshot

``user_data_callback`` attribute:

    Optional callback collecting user data (callable, default: None).
    See ``Snapshot.create()``.


Links
=====

Python issues:

* `#18874: Add a new tracemalloc module to trace Python
  memory allocations <http://bugs.python.org/issue18874>`_

Similar projects:

* `Meliae: Python Memory Usage Analyzer
  <https://pypi.python.org/pypi/meliae>`_
* `Issue #3329: API for setting the memory allocator used by Python
  <http://bugs.python.org/issue3329>`_
* `Guppy-PE: umbrella package combining Heapy and GSL
  <http://guppy-pe.sourceforge.net/>`_
* `PySizer <http://pysizer.8325.org/>`_: developed for Python 2.4
* `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_
* `pympler <http://code.google.com/p/pympler/>`_
* `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version of
  the CherryPy memory leak debugger
* `objgraph <http://mg.pov.lt/objgraph/>`_
* `caulk <https://github.com/smartfile/caulk/>`_

Copyright
=========

This document has been placed into the public domain.