diff --git a/pep-0454.txt b/pep-0454.txt index 22863ef05..25350c7ab 100644 --- a/pep-0454.txt +++ b/pep-0454.txt @@ -71,150 +71,69 @@ implementations of Python may not provide it. API === -To trace most memory blocks allocated by Python, the module should be -enabled as early as possible by calling ``tracemalloc.enable()`` -function, by setting the ``PYTHONTRACEMALLOC`` environment variable to -``1``, or by using ``-X tracemalloc`` command line option. +To trace most memory blocks allocated by Python, the module should be enabled +as early as possible by setting the ``PYTHONTRACEMALLOC`` environment +variable to ``1``, or by using ``-X tracemalloc`` command line +option. The ``tracemalloc.enable`` function can also be called to start +tracing Python memory allocations. -By default, the ``Trace.traceback`` attribute only stores one ``Frame`` -instance per allocated memory block. Use ``set_traceback_limit()`` to -store more frames. +By default, a trace of an allocated memory block only stores one frame. Use the +``set_traceback_limit`` function to store more frames. + +Python memory blocks allocated in the ``tracemalloc`` module are ignored by +default using a filter. Use the ``clear_filters`` function to see trace +also these memory allocations. + +At fork, the module is automatically disabled in the child process. -Functions ---------- +Main Functions +-------------- -``add_filter(filter)`` function: +``cancel_tasks()`` function: - Add a new filter on Python memory allocations, *filter* is a - ``Filter`` instance. + Cancel all scheduled tasks. - All inclusive filters are applied at once, a memory allocation is - only ignored if no inclusive filter match its trace. A memory - allocation is ignored if at least one exclusive filter matchs its - trace. - - The new filter is not applied on already collected traces. Use - ``clear_traces()`` to ensure that all traces match the new filter. - - -``add_include_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: - - Add an inclusive filter: helper for ``add_filter()`` creating a - ``Filter`` instance with ``include`` attribute set to ``True``. - - Example: ``tracemalloc.add_include_filter(tracemalloc.__file__)`` - only includes memory blocks allocated by the ``tracemalloc`` module. - - -``add_exclude_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: - - Add an exclusive filter: helper for ``add_filter()`` creating a - ``Filter`` instance with ``include`` attribute set to ``False``. - - Example: ``tracemalloc.add_exclude_filter(tracemalloc.__file__)`` - ignores memory blocks allocated by the ``tracemalloc`` module. - - -``clear_filters()`` function: - - Reset the filter list. + See also the ``get_tasks`` function. ``clear_traces()`` function: - Clear all traces and statistics on Python memory allocations, and - reset the ``get_traced_memory()`` counter. + Clear all traces and statistics on Python memory allocations, and reset the + ``get_arena_size`` and ``get_traced_memory`` counters. ``disable()`` function: - Stop tracing Python memory allocations and stop the timer started by - ``start_timer()``. + Stop tracing Python memory allocations and cancel scheduled tasks. - See also ``enable()`` and ``is_enabled()`` functions. + See also ``enable`` and ``is_enabled`` functions. ``enable()`` function: - Start tracing Python memory allocations. + Start tracing Python memory allocations. - See also ``disable()`` and ``is_enabled()`` functions. + At fork, the module is automatically disabled in the child process. - -``get_filters()`` function: - - Get the filters on Python memory allocations as list of ``Filter`` - instances. - - -``get_traceback_limit()`` function: - - Get the maximum number of ``Frame`` instances stored in the - ``traceback`` attribute of a ``Trace`` instance. - - Use ``set_traceback_limit()`` to change the limit. - - -``get_object_address(obj)`` function: - - Get the address of the memory block of the specified Python object. - - -``get_object_trace(obj)`` function: - - Get the trace of a Python object *obj* as a ``Trace`` instance. - - The function only returns the trace of the memory block directly - holding to object. The ``size`` attribute of the trace is smaller - than the total size of the object if the object is composed of more - than one memory block. - - Return ``None`` if the ``tracemalloc`` module did not trace the - allocation of the object. - - See also ``gc.get_referrers()`` and ``sys.getsizeof()`` functions. - - -``get_process_memory()`` function: - - Get the memory usage of the current process as a meminfo namedtuple - with two attributes: - - * ``rss``: Resident Set Size in bytes - * ``vms``: size of the virtual memory in bytes - - Return ``None`` if the platform is not supported. + See also ``disable`` and ``is_enabled`` functions. ``get_stats()`` function: - Get statistics on traced Python memory blocks as a dictionary - ``{filename (str): {line_number (int): stats}}`` where *stats* in a - ``TraceStats`` instance, *filename* and *line_number* can be - ``None``. + Get statistics on traced Python memory blocks as a dictionary ``{filename + (str): {line_number (int): stats}}`` where *stats* in a + ``(size: int, count: int)`` tuple, *filename* and *line_number* can + be ``None``. - Return an empty dictionary if the ``tracemalloc`` module is - disabled. + Return an empty dictionary if the ``tracemalloc`` module is disabled. + + See also the ``get_traces`` function. -``get_traced_memory()`` function: +``get_tasks()`` function: - Get the total size of all traced memory blocks allocated by Python. - - -``get_tracemalloc_size()`` function: - - Get the memory usage in bytes of the ``tracemalloc`` module. - - -``get_traces(obj)`` function: - - Get all traces of Python memory allocations as a dictionary - ``{address (int): trace}`` where *trace* is a ``Trace`` instance. - - Return an empty dictionary if the ``tracemalloc`` module is - disabled. + Get the list of scheduled tasks, list of ``Task`` instances. ``is_enabled()`` function: @@ -222,485 +141,864 @@ Functions ``True`` if the ``tracemalloc`` module is tracing Python memory allocations, ``False`` otherwise. - See also ``enable()`` and ``disable()`` functions. + See also ``enable`` and ``disable`` functions. -``start_timer(delay: int, func: callable, args: tuple=(), kwargs: dict={})`` function: +Trace Functions +--------------- - Start a timer calling ``func(*args, **kwargs)`` every *delay* - seconds. Enable the ``tracemalloc`` module if it is disabled. The - timer is based on the Python memory allocator, it is not real time. - *func* is called after at least *delay* seconds, it is not called - exactly after *delay* seconds if no Python memory allocation - occurred. The timer has a resolution of 1 second. +``get_traceback_limit()`` function: - If the ``start_timer()`` function is called twice, previous - parameters are replaced. Call the ``stop_timer()`` function to stop - the timer. + Get the maximum number of frames stored in the traceback of a trace of a + memory block. - The ``DisplayTopTask.start()`` and ``TakeSnapshot.start()`` methods - use the ``start_timer()`` function to run regulary a task. + Use the ``set_traceback_limit`` function to change the limit. -``set_traceback_limit(limit: int)`` function: +``get_object_address(obj)`` function: - Set the maximum number of ``Frame`` instances stored in the - ``traceback`` attribute of a ``Trace`` instance. Clear all traces - and statistics on Python memory allocations if the ``tracemalloc`` - module is enabled, + Get the address of the memory block of the specified Python object. - Storing the traceback of each memory allocation has an important - overhead on the memory usage. Example with the Python test suite: - tracing all memory allocations increases the memory usage by - ``+50%`` when storing only 1 frame and ``+150%`` when storing 10 - frames. Use ``get_tracemalloc_size()`` to measure the overhead and - ``add_filter()`` to select which memory allocations are traced. + A Python object can be composed by multiple memory blocks, the function only + returns the address of the main memory block. - Use ``get_traceback_limit()`` to get the current limit. + See also ``get_object_trace`` and ``gc.get_referrers`` functions. -``stop_timer()`` function: +``get_object_trace(obj)`` function: - Stop the timer started by ``start_timer()``. + Get the trace of a Python object *obj* as a ``(size: int, traceback)`` tuple + where *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, + *filename* and *lineno* can be ``None``. + + The function only returns the trace of the main memory block of the object. + The *size* of the trace is smaller than the total size of the object if the + object is composed by more than one memory block. + + Return ``None`` if the ``tracemalloc`` module did not trace the + allocation of the object. + + See also ``get_object_address``, ``get_trace``, ``get_traces``, + ``gc.get_referrers`` and ``sys.getsizeof`` functions. -DisplayTop class +``get_trace(address)`` function: + + Get the trace of a memory block as a ``(size: int, traceback)`` tuple where + *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, + *filename* and *lineno* can be ``None``. + + Return ``None`` if the ``tracemalloc`` module did not trace the + allocation of the memory block. + + See also ``get_object_trace``, ``get_stats`` and ``get_traces`` + functions. + + +``get_traces()`` function: + + Get all traces of Python memory allocations as a dictionary ``{address + (int): trace}`` where *trace* is a + ``(size: int, traceback)`` and *traceback* is a list of + ``(filename: str, lineno: int)``. + *traceback* can be empty, *filename* and *lineno* can be None. + + Return an empty dictionary if the ``tracemalloc`` module is disabled. + + See also ``get_object_trace``, ``get_stats`` and ``get_trace`` + functions. + + +``set_traceback_limit(nframe: int)`` function: + + Set the maximum number of frames stored in the traceback of a trace of a + memory block. + + Storing the traceback of each memory allocation has an important overhead on + the memory usage. Use the ``get_tracemalloc_memory`` function to measure + the overhead and the ``add_filter`` function to select which memory + allocations are traced. + + Use the ``get_traceback_limit`` function to get the current limit. + + +Filter Functions ---------------- +``add_filter(filter)`` function: + + Add a new filter on Python memory allocations, *filter* is a ``Filter`` + instance. + + All inclusive filters are applied at once, a memory allocation is only + ignored if no inclusive filter match its trace. A memory allocation is + ignored if at least one exclusive filter matchs its trace. + + The new filter is not applied on already collected traces. Use the + ``clear_traces`` function to ensure that all traces match the new + filter. + +``add_include_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: + + Add an inclusive filter: helper for the ``add_filter`` method creating a + ``Filter`` instance with the ``Filter.include`` attribute set to + ``True``. + + Example: ``tracemalloc.add_include_filter(tracemalloc.__file__)`` only + includes memory blocks allocated by the ``tracemalloc`` module. + + +``add_exclude_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: + + Add an exclusive filter: helper for the ``add_filter`` method creating a + ``Filter`` instance with the ``Filter.include`` attribute set to + ``False``. + + Example: ``tracemalloc.add_exclude_filter(tracemalloc.__file__)`` ignores + memory blocks allocated by the ``tracemalloc`` module. + + +``clear_filters()`` function: + + Reset the filter list. + + See also the ``get_filters`` function. + + +``get_filters()`` function: + + Get the filters on Python memory allocations as list of ``Filter`` + instances. + + See also the ``clear_filters`` function. + + +Metric Functions +---------------- + +The following functions can be used to add metrics to a snapshot, see +the ``Snapshot.add_metric`` method. + +``get_allocated_blocks()`` function: + + Get the current number of allocated memory blocks. + + +``get_arena_size()`` function: + + Get the size in bytes of traced arenas. + + See also the ``get_pymalloc_stats`` function. + + +``get_process_memory()`` function: + + Get the memory usage of the current process as a ``(rss: int, vms: int)`` + tuple, *rss* is the "Resident Set Size" in bytes and *vms* is the size of + the virtual memory in bytes + + Return ``None`` if the platform is not supported. + + +``get_pymalloc_stats()`` function: + + Get statistics on the ``pymalloc`` allocator as a dictionary. + + +---------------------+-------------------------------------------------------+ + | Key | Description | + +=====================+=======================================================+ + | ``alignment`` | Alignment of addresses returned to the user. | + +---------------------+-------------------------------------------------------+ + | ``threshold`` | Small block threshold in bytes: pymalloc uses | + | | PyMem_RawMalloc() for allocation greater than | + | | threshold. | + +---------------------+-------------------------------------------------------+ + | ``nalloc`` | Number of times object malloc called | + +---------------------+-------------------------------------------------------+ + | ``arena_size`` | Arena size in bytes | + +---------------------+-------------------------------------------------------+ + | ``total_arenas`` | Number of calls to new_arena(): total number of | + | | allocated arenas, including released arenas | + +---------------------+-------------------------------------------------------+ + | ``max_arenas`` | Maximum number of arenas | + +---------------------+-------------------------------------------------------+ + | ``arenas`` | Number of arenas currently allocated | + +---------------------+-------------------------------------------------------+ + | ``allocated_bytes`` | Number of bytes in allocated blocks | + +---------------------+-------------------------------------------------------+ + | ``available_bytes`` | Number of bytes in available blocks in used pools | + +---------------------+-------------------------------------------------------+ + | ``pool_size`` | Pool size in bytes | + +---------------------+-------------------------------------------------------+ + | ``free_pools`` | Number of unused pools | + +---------------------+-------------------------------------------------------+ + | ``pool_headers`` | Number of bytes wasted in pool headers | + +---------------------+-------------------------------------------------------+ + | ``quantization`` | Number of bytes in used and full pools wasted due to | + | | quantization, i.e. the necessarily leftover space at | + | | the ends of used and full pools. | + +---------------------+-------------------------------------------------------+ + | ``arena_alignment`` | Number of bytes for arena alignment padding | + +---------------------+-------------------------------------------------------+ + + The function is not available if Python is compiled without ``pymalloc``. + + See also ``get_arena_size`` and ``sys._debugmallocstats`` functions. + + +``get_traced_memory()`` function: + + Get the current size and maximum size of memory blocks traced by the + ``tracemalloc`` module as a tuple: ``(size: int, max_size: int)``. + + +``get_tracemalloc_memory()`` function: + + Get the memory usage in bytes of the ``tracemalloc`` module as a + tuple: ``(size: int, free: int)``. + + * *size*: total size of bytes allocated by the module, + including *free* bytes + * *free*: number of free bytes available to store data + + +``get_unicode_interned()`` function: + + Get the size in bytes and the length of the dictionary of Unicode interned + strings as a ``(size: int, length: int)`` tuple. + + +DisplayTop +---------- + ``DisplayTop()`` class: Display the top of allocated memory blocks. -``display_snapshot(snapshot, count=10, group_by="filename_lineno", cumulative=False, file=None)`` method: +``display(count=10, group_by="line", cumulative=False, file=None, callback=None)`` method: - Display a snapshot of memory blocks allocated by Python, *snapshot* - is a ``Snapshot`` instance. + Take a snapshot and display the top *count* biggest allocated memory + blocks grouped by *group_by*. + + *callback* is an optional callable object which can be used to add + metrics to a snapshot. It is called with only one parameter: the newly + created snapshot instance. Use the ``Snapshot.add_metric`` method to + add new metric. + + Return the snapshot, a ``Snapshot`` instance. + +``display_snapshot(snapshot, count=10, group_by="line", cumulative=False, file=None)`` method: + + Display a snapshot of memory blocks allocated by Python, *snapshot* is a + ``Snapshot`` instance. ``display_top_diff(top_diff, count=10, file=None)`` method: - Display differences between two ``GroupedStats`` instances, - *top_diff* is a ``StatsDiff`` instance. + Display differences between two ``GroupedStats`` instances, + *top_diff* is a ``StatsDiff`` instance. ``display_top_stats(top_stats, count=10, file=None)`` method: - Display the top of allocated memory blocks grouped by the - ``group_by`` attribute of *top_stats*, *top_stats* is a - ``GroupedStats`` instance. + Display the top of allocated memory blocks grouped by the + ``GroupedStats.group_by`` attribute of *top_stats*, *top_stats* is a + ``GroupedStats`` instance. + +``average`` attribute: + + If ``True`` (default value), display the average size of memory blocks. ``color`` attribute: - If ``True``, always use colors. If ``False``, never use colors. The - default value is ``None``: use colors if the *file* parameter is a - TTY device. + If ``True``, always use colors. If ``False``, never use colors. The + default value is ``None``: use colors if the *file* parameter is a TTY + device. -``compare_with_previous`` attribute: +``compare_to_previous`` attribute: - If ``True`` (default value), compare with the previous snapshot. If - ``False``, compare with the first snapshot. + If ``True`` (default value), compare to the previous snapshot. If + ``False``, compare to the first snapshot. + +``count`` attribute: + + If ``True`` (default value), display the number of allocated memory + blocks. ``filename_parts`` attribute: - Number of displayed filename parts (int, default: ``3``). Extra - parts are replaced with ``'...'``. + Number of displayed filename parts (int, default: ``3``). Extra parts + are replaced with ``'...'``. -``show_average`` attribute: +``metrics`` attribute: - If ``True`` (default value), display the average size of memory blocks. + If ``True`` (default value), display metrics: see + ``Snapshot.metrics``. -``show_count`` attribute: +``previous_top_stats`` attribute: - If ``True`` (default value), display the number of allocated memory - blocks. + Previous ``GroupedStats`` instance, or first ``GroupedStats`` + instance if ``compare_to_previous`` is ``False``, used to display the + differences between two snapshots. -``show_size`` attribute: +``size`` attribute: - If ``True`` (default value), display the size of memory blocks. + If ``True`` (default value), display the size of memory blocks. -DisplayTopTask class --------------------- +DisplayTopTask +-------------- -``DisplayTopTask(count=10, group_by="filename_lineno", cumulative=False, file=sys.stdout, user_data_callback=None)`` class: +``DisplayTopTask(count=10, group_by="line", cumulative=False, file=sys.stdout, callback=None)`` class: - Task taking temporary snapshots and displaying the top *count* - memory allocations grouped by *group_by*. + Task taking temporary snapshots and displaying the top *count* memory + allocations grouped by *group_by*. - Call the ``start()`` method to start the task. + ``DisplayTopTask`` is based on the ``Task`` class and so inherit + all attributes and methods, especially: + + * ``Task.cancel`` + * ``Task.schedule`` + * ``Task.set_delay`` + * ``Task.set_memory_threshold`` + + Modify the ``display_top`` attribute to customize the display. ``display()`` method: - Take a snapshot and display the top *count* biggest allocated memory - blocks grouped by *group_by* using the ``display_top`` attribute. + Take a snapshot and display the top ``count`` biggest allocated + memory blocks grouped by ``group_by`` using the ``display_top`` + attribute. - Return the snapshot, a ``Snapshot`` instance. + Return the snapshot, a ``Snapshot`` instance. -``start(delay: int)`` method: +``callback`` attribute: - Start a task using the ``start_timer()`` function calling the - ``display()`` method every *delay* seconds. - -``stop()`` method: - - Stop the task started by the ``start()`` method using the - ``stop_timer()`` function. + *callback* is an optional callable object which can be used to add + metrics to a snapshot. It is called with only one parameter: the newly + created snapshot instance. Use the ``Snapshot.add_metric`` method to + add new metric. ``count`` attribute: - Maximum number of displayed memory blocks. + Maximum number of displayed memory blocks. ``cumulative`` attribute: - If ``True``, cumulate size and count of memory blocks of all frames - of each ``Trace`` instance, not only the most recent frame. The - default value is ``False``. + If ``True``, cumulate size and count of memory blocks of all frames of + each trace, not only the most recent frame. The default value is + ``False``. - The option is ignored if the traceback limit is ``1``, see the - ``get_traceback_limit()`` function. + The option is ignored if the traceback limit is less than ``2``, see + the ``get_traceback_limit`` function. ``display_top`` attribute: - Instance of ``DisplayTop``. + Instance of ``DisplayTop``. ``file`` attribute: - The top is written into *file*. + The top is written into *file*. ``group_by`` attribute: - Determine how memory allocations are grouped: see - ``Snapshot.top_by`` for the available values. - -``user_data_callback`` attribute: - - Optional callback collecting user data (callable, default: - ``None``). See ``Snapshot.create()``. + Determine how memory allocations are grouped: see ``Snapshot.top_by`` + for the available values. -Filter class ------------- +Filter +------ ``Filter(include: bool, pattern: str, lineno: int=None, traceback: bool=False)`` class: - Filter to select which memory allocations are traced. Filters can be - used to reduce the memory usage of the ``tracemalloc`` module, which - can be read using ``get_tracemalloc_size()``. - -``match_trace(trace)`` method: - - Return ``True`` if the ``Trace`` instance must be kept according to - the filter, ``False`` otherwise. + Filter to select which memory allocations are traced. Filters can be used to + reduce the memory usage of the ``tracemalloc`` module, which can be read + using the ``get_tracemalloc_memory`` function. ``match(filename: str, lineno: int)`` method: - Return ``True`` if the filename and line number must be kept - according to the filter, ``False`` otherwise. + Return ``True`` if the filter matchs the filename and line number, + ``False`` otherwise. ``match_filename(filename: str)`` method: - Return ``True`` if the filename must be kept according to the - filter, ``False`` otherwise. + Return ``True`` if the filter matchs the filename, ``False`` otherwise. ``match_lineno(lineno: int)`` method: - Return ``True`` if the line number must be kept according to the - filter, ``False`` otherwise. + Return ``True`` if the filter matchs the line number, ``False`` + otherwise. + +``match_traceback(traceback)`` method: + + Return ``True`` if the filter matchs the *traceback*, ``False`` + otherwise. + + *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples. ``include`` attribute: - If *include* is ``True``, only trace memory blocks allocated in a - file with a name matching filename ``pattern`` at line number - ``lineno``. If *include* is ``False``, ignore memory blocks - allocated in a file with a name matching filename :attr`pattern` at - line number ``lineno``. + If *include* is ``True``, only trace memory blocks allocated in a file + with a name matching filename ``pattern`` at line number + ``lineno``. + + If *include* is ``False``, ignore memory blocks allocated in a file with + a name matching filename :attr`pattern` at line number ``lineno``. + +``lineno`` attribute: + + Line number (``int``). If is is ``None`` or less than ``1``, it matches + any line number. ``pattern`` attribute: - The filename *pattern* can contain one or many ``*`` joker - characters which match any substring, including an empty string. The - ``.pyc`` and ``.pyo`` suffixes are replaced with ``.py``. On - Windows, the comparison is case insensitive and the alternative - separator ``/`` is replaced with the standard separator ``\``. - -``lineno`` attribute: - - Line number (``int``). If is is ``None`` or lesser than ``1``, it - matches any line number. + The filename *pattern* can contain one or many ``*`` joker characters + which match any substring, including an empty string. The ``.pyc`` and + ``.pyo`` file extensions are replaced with ``.py``. On Windows, the + comparison is case insensitive and the alternative separator ``/`` is + replaced with the standard separator ``\``. ``traceback`` attribute: - If *traceback* is ``True``, all frames of the ``traceback`` - attribute of ``Trace`` instances are checked. If *traceback* is - ``False``, only the most recent frame is checked. + If *traceback* is ``True``, all frames of the traceback are checked. If + *traceback* is ``False``, only the most recent frame is checked. - This attribute only has an effect on the ``match_trace()`` method - and only if the traceback limit is greater than ``1``. See the - ``get_traceback_limit()`` function. + This attribute is ignored if the traceback limit is less than ``2``. + See the ``get_traceback_limit`` function. -Frame class ------------ +GroupedStats +------------ -``Frame`` class: +``GroupedStats(timestamp: datetime.datetime, stats: dict, group_by: str, cumulative=False, metrics: dict=None)`` class: - Trace of a Python frame, used by ``Trace.traceback`` attribute. + Top of allocated memory blocks grouped by *group_by* as a dictionary. -``filename`` attribute: - - Python filename, ``None`` if unknown. - -``lineno`` attribute: - - Python line number, ``None`` if unknown. - - -GroupedStats class ------------------- - -``GroupedStats(stats: dict, group_by: str, cumulative=False, timestamp=None, process_memory=None, tracemalloc_size=None)`` class: - - Top of allocated memory blocks grouped by on *group_by* as a - dictionary. - - The ``Snapshot.top_by()`` method creates a ``GroupedStats`` instance. + The ``Snapshot.top_by`` method creates a ``GroupedStats`` instance. ``compare_to(old_stats: GroupedStats=None)`` method: - Compare to an older ``GroupedStats`` instance. Return a - ``StatsDiff`` instance. + Compare to an older ``GroupedStats`` instance. + Return a ``StatsDiff`` instance. + + The ``StatsDiff.differences`` list is not sorted: call + the ``StatsDiff.sort`` method to sort the list. + + ``None`` values are replaced with an empty string for filenames or zero + for line numbers, because ``str`` and ``int`` cannot be + compared to ``None``. ``cumulative`` attribute: - If ``True``, cumulate size and count of memory blocks of all frames - of ``Trace``, not only the most recent frame. + If ``True``, cumulate size and count of memory blocks of all frames of + the traceback of a trace, not only the most recent frame. + +``metrics`` attribute: + + Dictionary storing metrics read when the snapshot was created: + ``{name (str): metric}`` where *metric* type is ``Metric``. ``group_by`` attribute: - Determine how memory allocations were grouped. The type of ``stats`` - keys depends on *group_by*: - - ===================== ======================== ============== - group_by description key type - ===================== ======================== ============== - ``'filename'`` filename ``str`` - ``'filename_lineno'`` filename and line number ``(str, str)`` - ``'address'`` memory block address ``int`` - ===================== ======================== ============== - - See the *group_by* parameter of the ``Snapshot.top_by()`` method. + Determine how memory allocations were grouped: see + ``Snapshot.top_by`` for the available values. ``stats`` attribute: - Dictionary ``{key: stats}`` where the *key* type depends on the - ``group_by`` attribute and *stats* type is ``TraceStats``. + Dictionary ``{key: stats}`` where the *key* type depends on the + ``group_by`` attribute and *stats* is a ``(size: int, count: int)`` + tuple. -``process_memory`` attribute: - - Result of the ``get_process_memory()`` function, can be ``None``. + See the ``Snapshot.top_by`` method. ``timestamp`` attribute: - Creation date and time of the snapshot, ``datetime.datetime`` - instance. - -``tracemalloc_size`` attribute: - - The memory usage in bytes of the ``tracemalloc`` module, result of - the ``get_tracemalloc_size()`` function. + Creation date and time of the snapshot, ``datetime.datetime`` + instance. -Snapshot class --------------- +Metric +------ -``Snapshot`` class: +``Metric(name: str, value: int, format: str)`` class: - Snapshot of memory blocks allocated by Python. + Value of a metric when a snapshot is created. + +``name`` attribute: + + Name of the metric. + +``value`` attribute: + + Value of the metric. + +``format`` attribute: + + Format of the metric: + + * ``int``: a number + * ``percent``: percentage (1.0 means 100%) + * ``size``: a size in bytes + + +Snapshot +-------- + +``Snapshot(timestamp: datetime.datetime, pid: int, traces: dict=None, stats: dict=None, metrics: dict=None)`` class: + + Snapshot of traces and statistics on memory blocks allocated by Python. + + Use ``TakeSnapshotTask`` to take regulary snapshots. + +``add_gc_metrics()`` method: + + Add a metric on garbage collector: + + * ``gc.objects``: total number of Python objects + + See the ``gc`` module. + + +``add_metric(name: str, value: int, format: str)`` method: + + Helper to add a ``Metric`` instance to ``Snapshot.metrics``. + Return the newly created ``Metric`` instance. + + Raise an exception if the name is already present in + ``Snapshot.metrics``. + + +``add_process_memory_metrics()`` method: + + Add metrics on the process memory: + + * ``process_memory.rss``: Resident Set Size + * ``process_memory.vms``: Virtual Memory Size + + These metrics are only available if the ``get_process_memory`` + function is available on the platform. + + +``add_pymalloc_metrics()`` method: + + Add metrics on the Python memory allocator (``pymalloc``): + + * ``pymalloc.blocks``: number of allocated memory blocks + * ``pymalloc.size``: size of ``pymalloc`` arenas + * ``pymalloc.max_size``: maximum size of ``pymalloc`` arenas + * ``pymalloc.allocated``: number of allocated bytes + * ``pymalloc.free``: number of free bytes + * ``pymalloc.fragmentation``: fragmentation percentage of the arenas + + These metrics are only available if Python is compiled in debug mode, + except ``pymalloc.blocks`` which is always available. + + +``add_tracemalloc_metrics()`` method: + + Add metrics on the ``tracemalloc`` module: + + * ``tracemalloc.traced.size``: size of memory blocks traced by the + ``tracemalloc`` module + * ``tracemalloc.traced.max_size``: maximum size of memory blocks traced + by the ``tracemalloc`` module + * ``tracemalloc.traces``: number of traces of Python memory blocks + * ``tracemalloc.module.size``: total size of bytes allocated by the + ``tracemalloc`` module, including free bytes + * ``tracemalloc.module.free``: number of free bytes available for + the ``tracemalloc`` module + * ``tracemalloc.module.fragmentation``: percentage of fragmentation of + the memory allocated by the ``tracemalloc`` module + * ``tracemalloc.arena_size``: size of traced arenas + + ``tracemalloc.traces`` metric is only present if the snapshot was created + with traces. + + +``add_unicode_metrics()`` method: + + Add metrics on the Unicode interned strings: + + * ``unicode_interned.size``: size of the dictionary, excluding size + of strings + * ``unicode_interned.len``: length of the dictionary - Use ``TakeSnapshot`` to take regulary snapshots. ``apply_filters(filters)`` method: - Apply a list filters on the ``traces`` and ``stats`` dictionaries, - *filters* is a list of ``Filter`` instances. + Apply filters on the ``traces`` and ``stats`` dictionaries, + *filters* is a list of ``Filter`` instances. -``create(\*, with_traces=False, with_stats=True, user_data_callback=None)`` classmethod: - Take a snapshot of traces and/or statistics of allocated memory - blocks. +``create(traces=False, metrics=True)`` classmethod: - If *with_traces* is ``True``, ``get_traces()`` is called and its - result is stored in the ``traces`` attribute. This attribute - contains more information than ``stats`` and uses more memory and - more disk space. If *with_traces* is ``False``, ``traces`` is set to - ``None``. + Take a snapshot of traces and/or statistics of allocated memory blocks. - If *with_stats* is ``True``, ``get_stats()`` is called and its - result is stored in the ``Snapshot.stats`` attribute. If - *with_stats* is ``False``, ``Snapshot.stats`` is set to ``None``. + If *traces* is ``True``, ``get_traces`` is called and its result + is stored in the ``Snapshot.traces`` attribute. This attribute + contains more information than ``Snapshot.stats`` and uses more + memory and more disk space. If *traces* is ``False``, + ``Snapshot.traces`` is set to ``None``. - *with_traces* and *with_stats* cannot be ``False`` at the same time. + If *metrics* is ``True``, fill ``Snapshot.metrics`` with metrics + using the following methods: - *user_data_callback* is an optional callable object. Its result - should be serializable by the ``pickle`` module, or - ``Snapshot.write()`` would fail. If *user_data_callback* is set, it - is called and the result is stored in the ``Snapshot.user_data`` - attribute. Otherwise, ``Snapshot.user_data`` is set to ``None``. + * ``add_gc_metrics`` + * ``add_process_memory_metrics`` + * ``add_pymalloc_metrics`` + * ``add_tracemalloc_metrics`` + * ``add_unicode_metrics`` - The ``tracemalloc`` module must be enabled to take a snapshot. See - the ``enable()`` function. + If *metrics* is ``False``, ``Snapshot.metrics`` is set to an empty + dictionary. -``load(filename)`` classmethod: + Tracebacks of traces are limited to ``traceback_limit`` frames. Call + ``set_traceback_limit`` before calling ``Snapshot.create`` to + store more frames. + + The ``tracemalloc`` module must be enabled to take a snapshot. See the + the ``enable`` function. + +``get_metric(name, default=None)`` method: + + Get the value of the metric called *name*. Return *default* if the metric + does not exist. + + +``load(filename, traces=True)`` classmethod: + + Load a snapshot from a file. + + If *traces* is ``False``, don't load traces. - Load a snapshot from a file. ``top_by(group_by: str, cumulative: bool=False)`` method: - Compute top statistics grouped by *group_by* as a ``GroupedStats`` - instance: + Compute top statistics grouped by *group_by* as a ``GroupedStats`` + instance: - ===================== ======================== ============== - group_by description key type - ===================== ======================== ============== - ``'filename'`` filename ``str`` - ``'filename_lineno'`` filename and line number ``(str, str)`` - ``'address'`` memory block address ``int`` - ===================== ======================== ============== + ===================== ======================== ============== + group_by description key type + ===================== ======================== ============== + ``'filename'`` filename ``str`` + ``'line'`` filename and line number ``(str, int)`` + ``'address'`` memory block address ``int`` + ===================== ======================== ============== + + If *cumulative* is ``True``, cumulate size and count of memory blocks of + all frames of the traceback of a trace, not only the most recent frame. + The *cumulative* parameter is ignored if *group_by* is ``'address'`` or + if the traceback limit is less than ``2``. - If *cumulative* is ``True``, cumulate size and count of memory - blocks of all frames of each ``Trace`` instance, not only the most - recent frame. The *cumulative* parameter is ignored if *group_by* is - ``'address'`` or if the traceback limit is ``1``. See the - ``traceback_limit`` attribute. ``write(filename)`` method: - Write the snapshot into a file. + Write the snapshot into a file. + + +``metrics`` attribute: + + Dictionary storing metrics read when the snapshot was created: + ``{name (str): metric}`` where *metric* type is ``Metric``. ``pid`` attribute: - Identifier of the process which created the snapshot, result of - ``os.getpid()``. - -``process_memory`` attribute: - - Memory usage of the current process, result of the - ``get_process_memory()`` function. It can be ``None``. + Identifier of the process which created the snapshot, result of + ``os.getpid``. ``stats`` attribute: - Statistics on traced Python memory, result of the ``get_stats()`` - function, if ``create()`` was called with *with_stats* equals to - ``True``, ``None`` otherwise. - -``tracemalloc_size`` attribute: - - The memory usage in bytes of the ``tracemalloc`` module, result of - the ``get_tracemalloc_size()`` function. + Statistics on traced Python memory, result of the ``get_stats`` + function. ``traceback_limit`` attribute: - The maximum number of frames stored in the ``traceback`` attribute - of a ``Trace``, result of the ``get_traceback_limit()`` function. + Maximum number of frames stored in a trace of a memory block allocated by + Python. ``traces`` attribute: - Traces of Python memory allocations, result of the ``get_traces()`` - function, if ``create()`` was called with *with_traces* equals to - ``True``, ``None`` otherwise. - - The ``traceback`` attribute of each ``Trace`` instance is limited to - ``traceback_limit`` frames. + Traces of Python memory allocations, result of the ``get_traces`` + function, can be ``None``. ``timestamp`` attribute: - Creation date and time of the snapshot, ``datetime.datetime`` - instance. - -``user_data`` attribute: - - Result of *user_data_callback* called in ``Snapshot.create()`` - (default: ``None``). + Creation date and time of the snapshot, ``datetime.datetime`` + instance. -StatsDiff class ---------------- +StatsDiff +--------- ``StatsDiff(differences, old_stats, new_stats)`` class: - Differences between two ``GroupedStats`` instances. By default, the - ``differences`` list is unsorted: call ``sort()`` to sort it. + Differences between two ``GroupedStats`` instances. - The ``GroupedStats.compare_to()`` method creates a ``StatsDiff`` - instance. + The ``GroupedStats.compare_to`` method creates a ``StatsDiff`` + instance. ``sort()`` method: - Sort the ``differences`` list from the biggest allocation to the - smallest. Sort by *size_diff*, *size*, *count_diff*, *count* and - then by *key*. + Sort the ``differences`` list from the biggest difference to the + smallest difference. Sort by ``abs(size_diff)``, *size*, + ``abs(count_diff)``, *count* and then by *key*. ``differences`` attribute: - Differences between ``old_stats`` and ``new_stats`` as a list of - ``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*, - *size*, *count_diff* and *count* are ``int``. The key type depends - on the ``group_by`` attribute of ``new_stats``: - - ===================== ======================== ============== - group_by description key type - ===================== ======================== ============== - ``'filename'`` filename ``str`` - ``'filename_lineno'`` filename and line number ``(str, str)`` - ``'address'`` memory block address ``int`` - ===================== ======================== ============== - - See the ``group_by`` attribute of the ``GroupedStats`` class. + Differences between ``old_stats`` and ``new_stats`` as a list of + ``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*, + *size*, *count_diff* and *count* are ``int``. The key type depends on the + ``GroupedStats.group_by`` attribute of ``new_stats``: see the + ``Snapshot.top_by`` method. ``old_stats`` attribute: - Old ``GroupedStats`` instance, can be ``None``. + Old ``GroupedStats`` instance, can be ``None``. ``new_stats`` attribute: - New ``GroupedStats`` instance. + New ``GroupedStats`` instance. -Trace class ------------ +Task +---- -``Trace`` class: +``Task(func, \*args, \*\*kw)`` class: - Debug information of a memory block allocated by Python. + Task calling ``func(*args, **kw)``. When scheduled, the task is called when + the traced memory is increased or decreased by more than *threshold* bytes, + or after *delay* seconds. -``size`` attribute: +``call()`` method: - Size in bytes of the memory block. - -``traceback`` attribute: - - Traceback where the memory block was allocated as a list of - ``Frame`` instances, most recent first. - - The list can be empty or incomplete if the ``tracemalloc`` module - was unable to retrieve the full traceback. - - The traceback is limited to ``get_traceback_limit()`` frames. Use - ``set_traceback_limit()`` to store more frames. + Call ``func(*args, **kw)`` and return the result. -TraceStats class +``cancel()`` method: + + Cancel the task. + + Do nothing if the task is not scheduled. + + +``get_delay()`` method: + + Get the delay in seconds. If the delay is ``None``, the timer is + disabled. + + +``get_memory_threshold()`` method: + + Get the threshold of the traced memory. When scheduled, the task is + called when the traced memory is increased or decreased by more than + *threshold* bytes. The memory threshold is disabled if *threshold* is + ``None``. + + See also the ``set_memory_threshold`` method and the + ``get_traced_memory`` function. + + +``schedule(repeat: int=None)`` method: + + Schedule the task *repeat* times. If *repeat* is ``None``, the task is + rescheduled after each call until it is cancelled. + + If the method is called twice, the task is rescheduled with the new + *repeat* parameter. + + The task must have a memory threshold or a delay: see ``set_delay`` + and ``set_memory_threshold`` methods. The ``tracemalloc`` must be + enabled to schedule a task: see the ``enable`` function. + + The task is cancelled if the ``call`` method raises an exception. + The task can be cancelled using the ``cancel`` method or the + ``cancel_tasks`` function. + + +``set_delay(seconds: int)`` method: + + Set the delay in seconds before the task will be called. Set the delay to + ``None`` to disable the timer. + + The timer is based on the Python memory allocator, it is not real time. + The task is called after at least *delay* seconds, it is not called + exactly after *delay* seconds if no Python memory allocation occurred. + The timer has a resolution of 1 second. + + The task is rescheduled if it was scheduled. + + +``set_memory_threshold(size: int)`` method: + + Set the threshold of the traced memory. When scheduled, the task is + called when the traced memory is increased or decreased by more than + *threshold* bytes. Set the threshold to ``None`` to disable it. + + The task is rescheduled if it was scheduled. + + See also the ``get_memory_threshold`` method and the + ``get_traced_memory`` function. + + +``func`` attribute: + + Function, callable object. + +``func_args`` attribute: + + Function arguments, ``tuple``. + +``func_kwargs`` attribute: + + Function keyword arguments, ``dict``. It can be ``None``. + + +TakeSnapshotTask ---------------- -``TraceStats`` class: +``TakeSnapshotTask(filename_template: str="tracemalloc-$counter.pickle", traces: bool=False, metrics: bool=True, callback: callable=None)`` class: - Statistics on Python memory allocations. + Task taking snapshots of Python memory allocations and writing them into + files. -``size`` attribute: + ``TakeSnapshotTask`` is based on the ``Task`` class and so inherit + all attributes and methods, especially: - Total size in bytes of allocated memory blocks. + * ``Task.cancel`` + * ``Task.schedule`` + * ``Task.set_delay`` + * ``Task.set_memory_threshold`` -``count`` attribute: +``take_snapshot()`` method: - Number of allocated memory blocks. + Take a snapshot and write it into a file. + Return ``(snapshot, filename)`` where *snapshot* is a ``Snapshot`` + instance and filename type is ``str``. + +``callback`` attribute: + + *callback* is an optional callable object which can be used to add + metrics to a snapshot. It is called with only one parameter: the newly + created snapshot instance. Use the ``Snapshot.add_metric`` method to + add new metric. + +``filename_template`` attribute: + + Template to create a filename. The template supports the following + variables: + + * ``$pid``: identifier of the current process + * ``$timestamp``: current date and time + * ``$counter``: counter starting at 1 and incremented at each snapshot, + formatted as 4 decimal digits + + The default template is ``tracemalloc-$counter.pickle``. + +``metrics`` attribute: + + Parameter passed to the ``Snapshot.create`` function. + +``traces`` attribute: + + Parameter passed to the ``Snapshot.create`` function. Links