PEP 558: Remove dynamic frame semantics proposal (#1051)

Changing the frame API semantics based on whether or not a
tracing function is active is tricky to implement and hard
to document clearly, so this simplifies the proposal by
instead having the frame API always expose a write-through
proxy at function scope, and restricting the dynamic
snapshot behaviour to the locals() builtin.

pep-0558.rst

@@ -6,7 +6,7 @@ Status: Draft
 Type: Standards Track
 Content-Type: text/x-rst
 Created: 2017-09-08
-Python-Version: 3.7
+Python-Version: 3.8
 Post-History: 2017-09-08
 
 
@@ -34,7 +34,7 @@ up to and including replication of local variable mutation bugs that
 can arise when a trace hook is installed [1]_.
 
 While this PEP considers CPython's current behaviour when no trace hooks are
-installed to be acceptable (and even desirable), it considers the current
+installed to be acceptable (and largely desirable), it considers the current
 behaviour when trace hooks are installed to be problematic, as it causes bugs
 like [1]_ *without* even reliably enabling the desired functionality of allowing
 debuggers like ``pdb`` to mutate local variables [3]_.
@@ -71,6 +71,10 @@ scope that bring the ``locals()`` builtin semantics closer to those used in
 regular operation, while also making the related frame API semantics clearer
 and easier for interactive debuggers to rely on.
 
+The proposed tracing mode changes also affect the semantics of frame object
+references obtained through other means, such as via a traceback, or via the
+``sys._getframe()`` API.
+
 
 New ``locals()`` documentation
 ------------------------------
@@ -259,40 +263,31 @@ defined ``locals()`` builtin, trace functions necessarily use the implementation
 dependent ``frame.f_locals`` interface, as a frame reference is what gets
 passed to hook implementations.
 
-In regular operation, nothing will change - ``frame.f_locals`` will be a direct
+Instead of being a direct reference to the dynamic snapshot returned by
-reference to the dynamic snapshot, and ``locals()`` will return a reference to
+``locals()``, ``frame.f_locals`` will be updated to instead return a dedicated
-that snapshot. This reflects the fact that it's only CPython's tracing mode
+proxy type (implemented as a private subclass of the existing
-semantics that are currently problematic.
-
-In tracing mode, however, we will change ``frame.f_locals`` to instead return
-a dedicated proxy type (implemented as a private subclass of the existing
 ``types.MappingProxyType``) that has two internal attributes not exposed as
 part of either the Python or public C API:
 
-* *mapping*: the dynamic snapshot that would be returned by ``frame.f_locals``
+* *mapping*: the dynamic snapshot that is returned by the ``locals()`` builtin
-  during regular operation
 * *frame*: the underlying frame that the snapshot is for
 
-The ``locals()`` builtin would be aware of this proxy type, and continue to
+``__setitem__`` and ``__delitem__`` operations on the proxy will affect not only
-return a reference to the dynamic snapshot even when in tracing mode.
+the dynamic snapshot, but *also* the corresponding fast local or cell reference
+on the underlying frame.
 
-As long as the process remains in tracing mode, then ``__setitem__`` and
+The ``locals()`` builtin will be made aware of this proxy type, and continue to
-``__delitem__`` operations on the proxy will affect not only the dynamic
+return a reference to the dynamic snapshot rather than to the write-through
-snapshot, but *also* the corresponding fast local or cell reference on the
+proxy.
-underlying frame.
-
-If the process leaves tracing mode (i.e. all previously installed trace hooks
-are uninstalled), then any already created proxy objects will remain in place,
-but their ``__setitem__`` and ``__delitem__`` methods will skip mutating
-the underlying frame.
 
 At the C API layer, ``PyEval_GetLocals()`` will implement the same semantics
 as the Python level ``locals()`` builtin, and a new ``PyFrame_GetLocals(frame)``
 accessor API will be provided to allow the proxy bypass logic to be encapsulated
 entirely inside the frame implementation. The C level equivalent of accessing
 ``pyframe.f_locals`` in Python will be to access ``cframe->f_locals`` directly
-(the one difference is that the Python descriptor will continue to include an
+(the one difference is that accessing ``pyframe.f_locals`` will continue to
-implicit snapshot refresh).
+implicitly refresh the dynamic snapshot, whereas C code will need to explicitly
+call ``PyFrame_GetLocals(frame)`` to refresh the snapshot).
 
 The ``PyFrame_LocalsToFast()`` function will be changed to always emit
 ``RuntimeError``, explaining that it is no longer a supported operation, and
@@ -300,42 +295,6 @@ affected code should be updated to rely on the write-through tracing mode
 proxy instead.
 
 
-Open Questions
-==============
-
-Is it necessary to restrict frame semantic changes to tracing mode?
--------------------------------------------------------------------
-
-It would be possible to say that ``frame.f_locals`` should *always* return a
-write-through proxy, even in regular operation.
-
-This PEP currently avoids that option for a couple of key reasons, one pragmatic
-and one more philosophical:
-
-* Object allocations and method wrappers aren't free, and tracing functions
-  aren't the only operations that access frame locals from outside the function.
-  Restricting the changes to tracing mode means that the additional memory and
-  execution time overhead of these changes are going to be as close to zero in
-  regular operation as we can possibly make them.
-* "Don't change what isn't broken": the current tracing mode problems are caused
-  by a requirement that's specific to tracing mode (support for external
-  rebinding of function local variable references), so it makes sense to also
-  restrict any related fixes to tracing mode
-
-The counter-argument to this is that it makes for a really subtle behavioural
-runtime state dependent distinction in how ``frame.f_locals`` works, and creates
-some new edge cases around how ``f_locals`` behaves as trace functions are added
-and removed. If the design adopted were instead "``frame.f_locals`` is always a
-write-through proxy, while ``locals()`` is always a dynamic snapshot", that
-would likely be both simpler to implement and easier to explain, suggesting
-that it may be a good idea.
-
-Regardless of how the CPython reference implementation chooses to handle this,
-optimising compilers and interpreters can potentially impose additional
-restrictions on debuggers, by making local variable mutation through frame
-objects an opt-in behaviour that may disable some optimisations.
-
-
 Design Discussion
 =================
 
@@ -377,6 +336,45 @@ the calling scope by default.
 There doesn't seem to be any reason for the PEP to change this.
 
 
+Changing the frame API semantics in regular operation
+-----------------------------------------------------
+
+Earlier versions of this PEP proposed having the semantics of the frame
+``f_locals`` attribute depend on whether or not a tracing hook was currently
+installed - only providing the write-through proxy behaviour when a tracing hook
+was active, and otherwise behaving the same as the ``locals()`` builtin.
+
+That was adopted as the original design proposal for a couple of key reasons,
+one pragmatic and one more philosophical:
+
+* Object allocations and method wrappers aren't free, and tracing functions
+  aren't the only operations that access frame locals from outside the function.
+  Restricting the changes to tracing mode meant that the additional memory and
+  execution time overhead of these changes would as close to zero in regular
+  operation as we can possibly make them.
+* "Don't change what isn't broken": the current tracing mode problems are caused
+  by a requirement that's specific to tracing mode (support for external
+  rebinding of function local variable references), so it made sense to also
+  restrict any related fixes to tracing mode
+
+However, actually attempting to implement and document that dynamic approach
+highlighted the fact that it makes for a really subtle runtime state dependent
+behaviour distinction in how ``frame.f_locals`` works, and creates several
+new edge cases around how ``f_locals`` behaves as trace functions are added
+and removed.
+
+Accordingly, the design was switched to the current one, where
+``frame.f_locals`` is always a write-through proxy, and ``locals()`` is always
+a dynamic snapshot, which is both simpler to implement and easier to explain.
+
+Regardless of how the CPython reference implementation chooses to handle this,
+optimising compilers and interpreters also remain free to impose additional
+restrictions on debuggers, by making local variable mutation through frame
+objects an opt-in behaviour that may disable some optimisations (just as the
+emulation of CPython's frame API is already an opt-in flag in some Python
+implementations).
+
+
 Historical semantics at function scope
 --------------------------------------