PEP 669: Add multiple tool and one-shot event handling capabilities (#2747)

pep-0669.rst

@@ -14,12 +14,12 @@ Abstract
 Using a profiler or debugger in CPython can have a severe impact on
 performance. Slowdowns by an order of magnitude are common.
 
-This PEP proposes an API for monitoring of Python programs running
+This PEP proposes an API for monitoring Python programs running
 on CPython that will enable monitoring at low cost.
 
 Although this PEP does not specify an implementation, it is expected that
 it will be implemented using the quickening step of
-:pep:`PEP 659 <659#quickening>`.
+:pep:`659`.
 
 A ``sys.monitoring`` namespace will be added, which will contain
 the relevant functions and enum.
@@ -44,9 +44,9 @@ the parts of the code that are modified and a relatively low cost to those
 parts that are modified. We can leverage this to provide an efficient
 mechanism for monitoring that was not possible in 3.10 or earlier.
 
-By using quickening, we expect that code run under a debugger on 3.11
+By using quickening, we expect that code run under a debugger on 3.12
-should easily outperform code run without a debugger on 3.10.
+should outperform code run without a debugger on 3.11.
-Profiling will still slow down execution, but by much less than in 3.10.
+Profiling will still slow down execution, but by much less than in 3.11.
 
 
 Specification
@@ -57,6 +57,9 @@ for events and by activating a set of events.
 
 Activating events and registering callback functions are independent of each other.
 
+Both registering callbacks and activating events are done on a per-tool basis.
+It is possible to have multiple tools that respond to different sets of events.
+
 Events
 ------
 
@@ -65,16 +68,16 @@ to tools. By activating events and by registering callback functions
 tools can respond to these events in any way that suits them.
 Events can be set globally, or for individual code objects.
 
-For 3.11, CPython will support the following events:
+For 3.12, CPython will support the following events:
 
-* PY_CALL: Call of a Python function (occurs immediately after the call, the callee's frame will be on the stack)
+* PY_START: Start of a Python function (occurs immediately after the call, the callee's frame will be on the stack)
 * PY_RESUME: Resumption of a Python function (for generator and coroutine functions), except for throw() calls.
 * PY_THROW: A Python function is resumed by a throw() call.
 * PY_RETURN: Return from a Python function (occurs immediately before the return, the callee's frame will be on the stack).
 * PY_YIELD: Yield from a Python function (occurs immediately before the yield, the callee's frame will be on the stack).
 * PY_UNWIND:  Exit from a Python function during exception unwinding.
-* C_CALL: Call of a builtin function (before the call in this case).
+* C_CALL: Call to any callable, except Python functions (before the call in this case).
-* C_RETURN: Return from a builtin function (after the return in this case).
+* C_RETURN: Return from any callable, except Python functions (after the return in this case).
 * RAISE: An exception is raised.
 * EXCEPTION_HANDLED: An exception is handled.
 * LINE: An instruction is about to be executed that has a different line number from the preceding instruction.
@@ -93,15 +96,48 @@ All events will be attributes of the ``Event`` enum in ``sys.monitoring``::
 Note that ``Event`` is an ``IntFlag`` which means that the events can be or-ed
 together to form a set of events.
 
+Tool identifiers
+----------------
+
+The VM can support up to 6 tools at once.
+Before registering or activating events, a tool should choose an identifier.
+Identifiers are integers in the range 0 to 5.
+
+::
+
+  sys.monitoring.use_tool_id(id)->None
+  sys.monitoring.free_tool_id(id)->None
+
+``sys.monitoring.use_tool_id`` raises a ``ValueError`` if ``id`` is in use.
+
+All IDs are treated the same by the VM with regard to events, but the following
+IDs are pre-defined to make co-operation of tools easier::
+
+  sys.monitoring.DEBUGGER_ID = 0
+  sys.monitoring.COVERAGE_ID = 1
+  sys.monitoring.PROFILER_ID = 2
+  sys.monitoring.OPTIMIZER_ID = 3
+
+There is no obligation to set an ID, nor is there anything preventing a tool from
+using an ID even it is already in use.
+However, tool are encouraged to use a unique ID and respect other tools.
+
+For example, if a debugger were attached and ``DEBUGGER_ID`` were in use, it should
+report an error, rather than carrying on regardless.
+
+The ``OPTIMIZER_ID`` is provided for tools like Cinder or PyTorch
+that want to optimize Python code, but need to decide what to
+optimize in a way that depends on some wider context.
+
 Setting events globally
 -----------------------
 
 Events can be controlled globally by modifying the set of events being monitored:
 
-* ``sys.monitoring.get_events()->Event``
+* ``sys.monitoring.get_events(tool_id:int)->Event``
   Returns the ``Event`` set for all the active events.
 
-* ``sys.monitoring.set_events(event_set: Event)``
+* ``sys.monitoring.set_events(tool_id:int, event_set: Event)``
   Activates all events which are set in ``event_set``.
 
 No events are active by default.
@@ -111,10 +147,10 @@ Per code object events
 
 Events can also be controlled on a per code object basis:
 
-* ``sys.monitoring.get_local_events(code: CodeType)->Event``
+* ``sys.monitoring.get_local_events(tool_id:int, code: CodeType)->Event``
   Returns the ``Event`` set for all the local events for ``code``
 
-* ``sys.monitoring.set_local_events(code: CodeType, event_set: Event)``
+* ``sys.monitoring.set_local_events(tool_id:int, code: CodeType, event_set: Event)``
   Activates all the local events for ``code``  which are set in ``event_set``.
 
 Local events add to global events, but do not mask them.
@@ -126,12 +162,12 @@ Register callback functions
 
 To register a callable for events call::
 
-  sys.monitoring.register_callback(event, func)
+  sys.monitoring.register_callback(tool_id:int, event: Event, func: Callable | None)
 
 ``register_callback`` returns the previously registered callback, or ``None``.
 
 Functions can be unregistered by calling
-``sys.monitoring.register_callback(event, None)``.
+``sys.monitoring.register_callback(tool_id, event, None)``.
 
 Callback functions can be registered and unregistered at any time.
 
@@ -145,23 +181,23 @@ Different events will provide the callback function with different arguments, as
 
 * All events starting with ``PY_``:
 
-    ``func(code: CodeType, instruction_offset: int)``
+    ``func(code: CodeType, instruction_offset: int) -> DISABLE | Any``
 
 * ``C_CALL`` and ``C_RETURN``:
 
-    ``func(code: CodeType, instruction_offset: int, callable: object)``
+    ``func(code: CodeType, instruction_offset: int, callable: object) -> DISABLE | Any``
 
 * ``RAISE`` and ``EXCEPTION_HANDLED``:
 
-    ``func(code: CodeType, instruction_offset: int, exception: BaseException)``
+    ``func(code: CodeType, instruction_offset: int, exception: BaseException) -> DISABLE | Any``
 
 * ``LINE``:
 
-    ``func(code: CodeType, line_number: int)``
+    ``func(code: CodeType, line_number: int) -> DISABLE | Any``
 
-* ``JUMP`` and ``BRANCH``:
+* ``BRANCH``:
 
-    ``func(code: CodeType, instruction_offset: int, destination_offset: int)``
+    ``func(code: CodeType, instruction_offset: int, destination_offset: int) -> DISABLE | Any``
 
   Note that the ``destination_offset`` is where the code will next execute.
   For an untaken branch this will be the offset of the instruction following
@@ -169,86 +205,169 @@ Different events will provide the callback function with different arguments, as
 
 * ``INSTRUCTION``:
 
-    ``func(code: CodeType, instruction_offset: int)``
+    ``func(code: CodeType, instruction_offset: int) -> DISABLE | Any``
 
 * ``MARKER``:
 
-    ``func(code: CodeType, instruction_offset: int, marker_id: int)``
+    ``func(code: CodeType, instruction_offset: int) -> DISABLE | Any``
+
+If a callback returns ``sys.monitoring.DISABLE`` then that tool will not
+recieve any more events for that ``(code, instruction_offset)``.
+
+This feature is provided for coverage and other tools that are only interested
+seeing an event once. 
+
+Tools may see events after returning ``DISABLE``, in which case, they will not see
+those events until ``sys.monitoring.restart_events()`` is called.
+Note that ``sys.monitoring.restart_events()`` is not specific to one tool,
+so tools must be prepared to recieve events that they have chosen to DISABLE.
+
+Events in callback functions
+----------------------------
+
+Events are suspended in callback functions and their callees for the tool
+that registered that callback.
+
+That means that other tools will see events in the callback functions for other
+tools. This could be useful for debugging a profiling tool, but would produce
+misleading profiles, as the debugger tool would show up in the profile.
 
 Inserting and removing markers
-''''''''''''''''''''''''''''''''''
+------------------------------
 
 Two new functions are added to the ``sys`` module to support markers.
 
-* ``sys.monitoring.insert_marker(code: CodeType, offset: int, marker_id=0: range(256))``
+* ``sys.monitoring.insert_marker(tool_id: int, code: CodeType, offset: int)``
-* ``sys.monitoring.remove_marker(code: CodeType, offset: int)``
+* ``sys.monitoring.remove_marker(tool_id: int, code: CodeType, offset: int)``
 
-The ``marker_id`` has no meaning to the VM,
+A single code object may not have more than 255 markers at once.
-and is used only as an argument to the callback function.
+``sys.monitoring.insert_marker`` raises a ``ValueError`` if this limit
-The ``marker_id`` must in the range 0 to 255 (inclusive).
+is exceeded.
+
+Order of events
+---------------
+
+If an instructions triggers several events the occur in the following order:
+
+* MARKER
+* INSTRUCTION
+* LINE
+* All other events (only one of these event can occur per instruction)
+
+Each event is delivered to tools in ascending order of ID.
 
 Attributes of the ``sys.monitoring`` namespace
-''''''''''''''''''''''''''''''''''''''''''''''
+----------------------------------------------
 
 * ``class Event(enum.IntFlag)``
-* ``def get_events()->Event``
+* ``def use_tool_id(id)->None``
-* ``def set_events(event_set: Event)->None``
+* ``def free_tool_id(id)->None``
-* ``def get_local_events(code: CodeType)->Event``
+* ``def get_events(tool_id: int)->Event``
-* ``def set_local_events(code: CodeType, event_set: Event)->None``
+* ``def set_events(tool_id: int, event_set: Event)->None``
-* ``def register_callback(event: Event, func: Callable)->Optional[Callable]``
+* ``def get_local_events(tool_id: int, code: CodeType)->Event``
-* ``def insert_marker(code: CodeType, offset: Event, marker_id=0: range(256))->None``
+* ``def set_local_events(tool_id: int, code: CodeType, event_set: Event)->None``
-* ``def remove_marker(code: CodeType, offset: Event)->None``
+* ``def register_callback(tool_id: int, event: Event, func: Callable)->Optional[Callable]``
+* ``def insert_marker(tool_id: int, code: CodeType, offset: Event)->None``
+* ``def remove_marker(tool_id: int, code: CodeType, offset: Event)->None``
+* ``def restart_events()->None``
+* ``DISABLE: object``
+
 
 Backwards Compatibility
 =======================
 
-This PEP is fully backwards compatible, in the sense that old code 
+This PEP is mostly backwards compatible.
-will work if the features of this PEP are unused.
 
-However, if it is used it will effectively disable ``sys.settrace``, 
+This PEP is incompatible with :pep:`523` as the behavior would be undefined,
-``sys.setprofile`` and :pep:`523` frame evaluation.
+as we have no control over the behavior of :pep:`523` plugins.
 
-If :pep:`523` is in use, or ``sys.settrace`` or ``sys.setprofile`` has been
+Thus, if :pep:`523` is in use, then calling ``sys.monitoring.set_events()`` or
-set, then calling ``sys.monitoring.set_events()`` or
 ``sys.monitoring.set_local_events()`` will raise an exception.
 
 Likewise, if ``sys.monitoring.set_events()`` or
 ``sys.monitoring.set_local_events()`` has been called, then using :pep:`523`
-or calling ``sys.settrace`` or ``sys.setprofile`` will raise an exception.
+will raise an exception.
 
-This PEP is incompatible with ``sys.settrace`` and ``sys.setprofile``
+``sys.settrace`` and ``sys.setprofile`` will act as if they were tools 6 and 7
-because the implementation of ``sys.settrace`` and ``sys.setprofile``
+respectively, so can be used along side this PEP.
-will use the same underlying mechanism as this PEP. It would be too slow
-to support both the new and old monitoring mechanisms at the same time,
-and they would interfere in awkward ways if both were active at the same time.
 
-This PEP is incompatible with :pep:`523`, because :pep:`523` prevents the VM being
+This makes ``sys.settrace`` and ``sys.setprofile`` incompatible with :pep:`523`.
-able to modify the code objects of executing code, which is a necessary feature.
+Arguably, they already were as the author know of any PEP 523 plugin that support
-
+``sys.settrace`` or ``sys.setprofile`` correctly. This PEP merely formalizes that.
-We may seek to remove ``sys.settrace`` and :pep:`523` in the future once the APIs
-provided by this PEP have been widely adopted, but that is for another PEP.
 
 Performance
 -----------
 
-If no events are active, this PEP should have a negligible impact on
+If no events are active, this PEP should have a small positive impact on
-performance. 
+performance. Experiments show between 1 and 2% speedup from not supporting
+``sys.settrace()`` directly.
+
+The performance of ``sys.settrace()`` will be worse.
+The performance of ``sys.setprofile()`` should be better.
+However, by the tools relying on ``sys.settrace()`` and ``sys.setprofile()`` 
+can be made a lot faster by using the API provided by this PEP.
 
 If a small set of events are active, e.g. for a debugger, then the overhead
 of callbacks will be orders of magnitudes less than for ``sys.settrace`` and 
 much cheaper than using :pep:`523`.
 
+Coverage tools can be implemented at very low cost,
+by returning ``DISABLE`` in all callbacks.
+
 For heavily instrumented code, e.g. using ``LINE``, performance should be
 better than ``sys.settrace``, but not by that much as performance will be
 dominated by the time spent in callbacks.
 
 For optimizing virtual machines, such as future versions of CPython
-(and ``PyPy`` should they choose to support this API), changing the set of
+(and ``PyPy`` should they choose to support this API), changes to the set
-globally active events in the midst of a long running program could be quite
+active events in the midst of a long running program could be quite
 expensive, possibly taking hundreds of milliseconds as it triggers
 de-optimizations. Once such de-optimization has occurred, performance should
 recover as the VM can re-optimize the instrumented code.
 
+In general these operations can be considered to be fast:
+
+* ``def get_events(tool_id: int)->Event``
+* ``def get_local_events(tool_id: int, code: CodeType)->Event``
+* ``def register_callback(tool_id: int, event: Event, func: Callable)->Optional[Callable]``
+
+These operations are slower, but not especially so:
+
+* ``def set_local_events(tool_id: int, code: CodeType, event_set: Event)->None``
+* ``def insert_marker(tool_id: int, code: CodeType, offset: Event)->None``
+* ``def remove_marker(tool_id: int, code: CodeType, offset: Event)->None``
+
+And these operations should be regarded as slow:
+
+* ``def use_tool_id(id)->None``
+* ``def free_tool_id(id)->None``
+* ``def set_events(tool_id: int, event_set: Event)->None``
+* ``def restart_events()->None``
+
+How slow, the slow operation the operations are, depends on when then happen.
+If done early in the program, before modules are loaded,
+they should be fairly inexpensive.
+
+Memory Consumption
+''''''''''''''''''
+
+When not in use, this PEP will have a neglible change on memory consumption.
+
+How memory is used is very much an implementation detail.
+However, we expect that for 3.12 the additional memory consumption per
+code object will be **roughly** as follows:
+
++-------------+--------+--------+-------------+
+|                      |   Events             |
++-------------+--------+--------+-------------+
+|    Tools    | Others |  LINE  | INSTRUCTION |
++=============+========+========+=============+
+|      One    | None   |  ≈40%  |    ≈80%     |
++-------------+--------+--------+-------------+
++ Two or more |  ≈40%  | ≈120%  |    ≈200%    |
++-------------+--------+--------+-------------+
+
+
 Security Implications
 =====================
 
@@ -269,8 +388,9 @@ step of :pep:`PEP 659 <659#quickening>`.
 Activating some events will cause all code objects to
 be quickened before they are executed.
 
-For example, if the ``LINE`` event is turned on, then all instructions that
+For example, if the ``C_CALL`` event is turned on,
-are at the start of a line will be replaced with a ``LINE_EVENT`` instruction.
+then all call instructions will be
+replaced with a ``INSTRUMENTED_CALL`` instruction.
 
 Note that this will interfere with specialization, which will result in some
 performance degradation in addition to the overhead of calling the
@@ -289,7 +409,7 @@ underlying event occurs.
 The exact set of events that require instrumentation is an implementation detail,
 but for the current design, the following events will require instrumentation:
 
-* PY_CALL
+* PY_START
 * PY_RESUME
 * PY_RETURN
 * PY_YIELD
@@ -300,6 +420,13 @@ but for the current design, the following events will require instrumentation:
 * JUMP
 * BRANCH
 
+Each instrumented bytecode will require an additional 8 bits of information to
+note which tool the instrumentation applies to.
+``LINE`` and ``INSTRUCTION`` events require additional information, as they
+need to store the original instruction, or even the instrumented instruction
+if they overlap other instrumentation.
+
+
 Implementing tools
 ==================
 
@@ -351,7 +478,6 @@ Debuggers can use the ``PY_CALL``, etc. events to be informed when
 a code object is first encountered, so that any necessary breakpoints
 can be inserted.
 
-
 Coverage Tools
 --------------
 
@@ -383,7 +509,7 @@ Line based profilers
 
 Line based profilers can use the ``LINE`` and ``JUMP`` events.
 Implementers of profilers should be aware that instrumenting ``LINE``
-and ``JUMP`` events will have a large impact on performance.
+events will have a large impact on performance.
 
 .. note::