python-peps/peps/pep-0558.rst

PEP: 558
Title: Defined semantics for locals()
Author: Alyssa Coghlan <ncoghlan@gmail.com>
BDFL-Delegate: Nathaniel J. Smith
Discussions-To: python-dev@python.org
Status: Withdrawn
Type: Standards Track
Content-Type: text/x-rst
Created: 08-Sep-2017
Python-Version: 3.13
Post-History: 08-Sep-2017, 22-May-2019, 30-May-2019, 30-Dec-2019, 18-Jul-2021,
              26-Aug-2021

PEP Withdrawal
==============

In December 2021, this PEP and :pep:`667` converged on a common definition of the
proposed changes to the Python level semantics of the ``locals()`` builtin (as
documented in the PEP text below), with the only remaining differences being
in the proposed C API changes and various internal implementation details.

Of those remaining differences, the most significant one was that :pep:`667`
at the time still proposed an immediate backwards compatibility break for the
``PyEval_GetLocals()`` API as soon as the PEP was accepted and implemented.

:pep:`667` has since been changed to propose a generous deprecation period for
the ``PyEval_GetLocals()`` API, continuing to support it in parallel with the
improved semantics offered by the new ``PyEval_GetFrameLocals()`` API.

Any remaining C API design concerns relate to new informational APIs that can be
added at a later date if they are deemed necessary, and any potential concerns
about the exact performance characteristics of the frame locals view implementation
are outweighed by the availability of a viable reference implementation.

Accordingly, this PEP has been withdrawn in favour of proceeding with :pep:`667`.

Note: while implementing :pep:`667` it became apparent that the rationale for and impact
of ``locals()`` being updated to return independent snapshots in
:term:`optimized scopes <py3.13:optimized scope>` was not entirely clear in either PEP.
The Motivation and Rationale sections in this PEP have been updated accordingly (since those
aspects are equally applicable to the accepted :pep:`667`).

Abstract
========

The semantics of the ``locals()`` builtin have historically been underspecified
and hence implementation dependent.

This PEP proposes formally standardising on the behaviour of the CPython 3.10
reference implementation for most execution scopes, with some adjustments to the
behaviour at function scope to make it more predictable and independent of the
presence or absence of tracing functions.

In addition, it proposes that the following functions be added to the stable
Python C API/ABI:

.. code-block:: c

    typedef enum {
      PyLocals_UNDEFINED = -1,
      PyLocals_DIRECT_REFERENCE = 0,
      PyLocals_SHALLOW_COPY = 1,
      _PyLocals_ENSURE_32BIT_ENUM = 2147483647
    } PyLocals_Kind;

    PyLocals_Kind PyLocals_GetKind();
    PyObject * PyLocals_Get();
    PyObject * PyLocals_GetCopy();

It also proposes the addition of several supporting functions and type
definitions to the CPython C API.

.. _pep-558-motivation:

Motivation
==========

While the precise semantics of the ``locals()`` builtin are nominally undefined,
in practice, many Python programs depend on it behaving exactly as it behaves in
CPython (at least when no tracing functions are installed).

Other implementations such as PyPy are currently replicating that behaviour,
up to and including replication of local variable mutation bugs that
can arise when a trace hook is installed [1]_.

While this PEP considers CPython's current behaviour when no trace hooks are
installed to be largely acceptable, it considers the current
behaviour when trace hooks are installed to be problematic, as it causes bugs
like [1]_ *without* even reliably enabling the desired functionality of allowing
debuggers like ``pdb`` to mutate local variables [3]_.

Review of the initial PEP and the draft implementation then identified an
opportunity for simplification of both the documentation and implementation
of the function level ``locals()`` behaviour by updating it to return an
independent snapshot of the function locals and closure variables on each
call, rather than continuing to return the semi-dynamic intermittently updated
shared copy that it has historically returned in CPython.

Specifically, the proposal in this PEP eliminates the historical behaviour where
adding a new local variable can change the behaviour of code executed with
``exec()`` in function scopes, even if that code runs *before* the local variable
is defined.

For example::

    def f():
        exec("x = 1")
        print(locals().get("x"))
    f()

prints ``1``, but::

    def f():
        exec("x = 1")
        print(locals().get("x"))
        x = 0
    f()

prints ``None`` (the default value from the ``.get()`` call).

With this PEP both examples would print ``None``, as the call to
``exec()`` and the subsequent call to ``locals()`` would use
independent dictionary snapshots of the local variables rather
than using the same shared dictionary cached on the frame object.

Proposal
========

The expected semantics of the ``locals()`` builtin change based on the current
execution scope. For this purpose, the defined scopes of execution are:

* module scope: top-level module code, as well as any other code executed using
  ``exec()`` or ``eval()`` with a single namespace
* class scope: code in the body of a ``class`` statement, as well as any other
  code executed using ``exec()`` or ``eval()`` with separate local and global
  namespaces
* function scope: code in the body of a ``def`` or ``async def`` statement,
  or any other construct that creates an optimized code block in CPython (e.g.
  comprehensions, lambda functions)

This PEP proposes elevating most of the current behaviour of the CPython
reference implementation to become part of the language specification, *except*
that each call to ``locals()`` at function scope will create a new dictionary
object, rather than caching a common dict instance in the frame object that
each invocation will update and return.

This PEP also proposes to largely eliminate the concept of a separate "tracing"
mode from the CPython reference implementation. In releases up to and including
Python 3.10, the CPython interpreter behaves differently when a trace hook has
been registered in one or more threads via an implementation dependent mechanism
like ``sys.settrace`` ([4]_) in CPython's ``sys`` module or
``PyEval_SetTrace`` ([5]_) in CPython's C API. If this PEP is accepted, then
the only remaining behavioural difference when a trace hook is installed is that
some optimisations in the interpreter eval loop are disabled when the tracing
logic needs to run after each opcode.

This PEP proposes changes to CPython's behaviour at function scope that make
the ``locals()`` builtin semantics when a trace hook is registered identical to
those used when no trace hook is registered, while also making the related frame
API semantics clearer and easier for interactive debuggers to rely on.

The proposed elimination of tracing mode affects the semantics of frame object
references obtained through other means, such as via a traceback, or via the
``sys._getframe()`` API, as the write-through semantics needed for trace hook
support are always provided by the ``f_locals`` attribute on frame objects,
rather than being runtime state dependent.


New ``locals()`` documentation
------------------------------

The heart of this proposal is to revise the documentation for the ``locals()``
builtin to read as follows:

    Return a mapping object representing the current local symbol table, with
    variable names as the keys, and their currently bound references as the
    values.

    At module scope, as well as when using ``exec()`` or ``eval()`` with a
    single namespace, this function returns the same namespace as ``globals()``.

    At class scope, it returns the namespace that will be passed to the
    metaclass constructor.

    When using ``exec()`` or ``eval()`` with separate local and global
    namespaces, it returns the local namespace passed in to the function call.

    In all of the above cases, each call to ``locals()`` in a given frame of
    execution will return the *same* mapping object. Changes made through
    the mapping object returned from ``locals()`` will be visible as bound,
    rebound, or deleted local variables, and binding, rebinding, or deleting
    local variables will immediately affect the contents of the returned mapping
    object.

    At function scope (including for generators and coroutines), each call to
    ``locals()`` instead returns a fresh dictionary containing the current
    bindings of the function's local variables and any nonlocal cell references.
    In this case, name binding changes made via the returned dict are *not*
    written back to the corresponding local variables or nonlocal cell
    references, and binding, rebinding, or deleting local variables and nonlocal
    cell references does *not* affect the contents of previously returned
    dictionaries.


There would also be a ``versionchanged`` note for the release making this change:

    In prior versions, the semantics of mutating the mapping object returned
    from ``locals()`` were formally undefined. In CPython specifically,
    the mapping returned at function scope could be implicitly refreshed by
    other operations, such as calling ``locals()`` again, or the interpreter
    implicitly invoking a Python level trace function. Obtaining the legacy
    CPython behaviour now requires explicit calls to update the initially
    returned dictionary with the results of subsequent calls to ``locals()``.


For reference, the current documentation of this builtin reads as follows:

    Update and return a dictionary representing the current local symbol table.
    Free variables are returned by locals() when it is called in function
    blocks, but not in class blocks.

    Note: The contents of this dictionary should not be modified; changes may
    not affect the values of local and free variables used by the interpreter.

(In other words: the status quo is that the semantics and behaviour of
``locals()`` are formally implementation defined, whereas the proposed
state after this PEP is that the only implementation defined behaviour will be
that associated with whether or not the implementation emulates the CPython
frame API, with the behaviour in all other cases being defined by the language
and library references)


Module scope
------------

At module scope, as well as when using ``exec()`` or ``eval()`` with a
single namespace, ``locals()`` must return the same object as ``globals()``,
which must be the actual execution namespace (available as
``inspect.currentframe().f_locals`` in implementations that provide access
to frame objects).

Variable assignments during subsequent code execution in the same scope must
dynamically change the contents of the returned mapping, and changes to the
returned mapping must change the values bound to local variable names in the
execution environment.

To capture this expectation as part of the language specification, the following
paragraph will be added to the documentation for ``locals()``:

   At module scope, as well as when using ``exec()`` or ``eval()`` with a
   single namespace, this function returns the same namespace as ``globals()``.

This part of the proposal does not require any changes to the reference
implementation - it is standardisation of the current behaviour.


Class scope
-----------

At class scope, as well as when using ``exec()`` or ``eval()`` with separate
global and local namespaces, ``locals()`` must return the specified local
namespace (which may be supplied by the metaclass ``__prepare__`` method
in the case of classes). As for module scope, this must be a direct reference
to the actual execution namespace (available as
``inspect.currentframe().f_locals`` in implementations that provide access
to frame objects).

Variable assignments during subsequent code execution in the same scope must
change the contents of the returned mapping, and changes to the returned mapping
must change the values bound to local variable names in the
execution environment.

The mapping returned by ``locals()`` will *not* be used as the actual class
namespace underlying the defined class (the class creation process will copy
the contents to a fresh dictionary that is only accessible by going through the
class machinery).

For nested classes defined inside a function, any nonlocal cells referenced from
the class scope are *not* included in the ``locals()`` mapping.

To capture this expectation as part of the language specification, the following
two paragraphs will be added to the documentation for ``locals()``:

   When using ``exec()`` or ``eval()`` with separate local and global
   namespaces, [this function] returns the given local namespace.

   At class scope, it returns the namespace that will be passed to the metaclass
   constructor.

This part of the proposal does not require any changes to the reference
implementation - it is standardisation of the current behaviour.


Function scope
--------------

At function scope, interpreter implementations are granted significant freedom
to optimise local variable access, and hence are NOT required to permit
arbitrary modification of local and nonlocal variable bindings through the
mapping returned from ``locals()``.

Historically, this leniency has been described in the language specification
with the words "The contents of this dictionary should not be modified; changes
may not affect the values of local and free variables used by the interpreter."

This PEP proposes to change that text to instead say:

    At function scope (including for generators and coroutines), each call to
    ``locals()`` instead returns a fresh dictionary containing the current
    bindings of the function's local variables and any nonlocal cell references.
    In this case, name binding changes made via the returned dict are *not*
    written back to the corresponding local variables or nonlocal cell
    references, and binding, rebinding, or deleting local variables and nonlocal
    cell references does *not* affect the contents of previously returned
    dictionaries.

This part of the proposal *does* require changes to the CPython reference
implementation, as CPython currently returns a shared mapping object that may
be implicitly refreshed by additional calls to ``locals()``, and the
"write back" strategy currently used to support namespace changes
from trace functions also doesn't comply with it (and causes the quirky
behavioural problems mentioned in the Motivation above).


CPython Implementation Changes
==============================

Summary of proposed implementation-specific changes
---------------------------------------------------

* Changes are made as necessary to provide the updated Python level semantics
* Two new functions are added to the stable ABI to replicate the updated
  behaviour of the Python ``locals()`` builtin:

.. code-block:: c

     PyObject * PyLocals_Get();
     PyLocals_Kind PyLocals_GetKind();

* One new function is added to the stable ABI to efficiently get a snapshot of
  the local namespace in the running frame:

.. code-block:: c

     PyObject * PyLocals_GetCopy();

* Corresponding frame accessor functions for these new public APIs are added to
  the CPython frame C API
* On optimised frames, the Python level ``f_locals`` API will return dynamically
  created read/write proxy objects that directly access the frame's local and
  closure variable storage. To provide interoperability with the existing
  ``PyEval_GetLocals()`` API, the proxy objects will continue to use the C level
  frame locals data storage field to hold a value cache that also allows for
  storage of arbitrary additional keys. Additional details on the expected
  behaviour of these fast locals proxy objects are covered below.
* No C API function is added to get access to a mutable mapping for the local
  namespace. Instead, ``PyObject_GetAttrString(frame, "f_locals")`` is used, the
  same API as is used in Python code.
* ``PyEval_GetLocals()`` remains supported and does not emit a programmatic
  warning, but will be deprecated in the documentation in favour of the new
  APIs that don't rely on returning a borrowed reference
* ``PyFrame_FastToLocals()`` and ``PyFrame_FastToLocalsWithError()`` remain
  supported and do not emit a programmatic warning, but will be deprecated in
  the documentation in favour of the new APIs that don't require direct access
  to the internal data storage layout of frame objects
* ``PyFrame_LocalsToFast()`` always raises ``RuntimeError()``, indicating that
  ``PyObject_GetAttrString(frame, "f_locals")`` should be used to obtain a
  mutable read/write mapping for the local variables.
* The trace hook implementation will no longer call ``PyFrame_FastToLocals()``
  implicitly. The version porting guide will recommend migrating to
  ``PyFrame_GetLocals()`` for read-only access and
  ``PyObject_GetAttrString(frame, "f_locals")`` for read/write access.


Providing the updated Python level semantics
--------------------------------------------

The implementation of the ``locals()`` builtin is modified to return a distinct
copy of the local namespace for optimised frames, rather than a direct reference
to the internal frame value cache updated by the ``PyFrame_FastToLocals()`` C
API and returned by the ``PyEval_GetLocals()`` C API.


Resolving the issues with tracing mode behaviour
------------------------------------------------

The current cause of CPython's tracing mode quirks (both the side effects from
simply installing a tracing function and the fact that writing values back to
function locals only works for the specific function being traced) is the way
that locals mutation support for trace hooks is currently implemented: the
``PyFrame_LocalsToFast`` function.

When a trace function is installed, CPython currently does the following for
function frames (those where the code object uses "fast locals" semantics):

1. Calls ``PyFrame_FastToLocals`` to update the frame value cache
2. Calls the trace hook (with tracing of the hook itself disabled)
3. Calls ``PyFrame_LocalsToFast`` to capture any changes made to the frame
   value cache

This approach is problematic for a few different reasons:

* Even if the trace function doesn't mutate the value cache, the final step
  resets any cell references back to the state they were in before the trace
  function was called (this is the root cause of the bug report in [1]_)
* If the trace function *does* mutate the value cache, but then does something
  that causes the value cache to be refreshed from the frame, those changes are
  lost (this is one aspect of the bug report in [3]_)
* If the trace function attempts to mutate the local variables of a frame other
  than the one being traced (e.g. ``frame.f_back.f_locals``), those changes
  will almost certainly be lost (this is another aspect of the bug report in
  [3]_)
* If a reference to the frame value cache (e.g. retrieved via ``locals()``) is
  passed to another function, and *that* function mutates the value cache, then
  those changes *may* be written back to the execution frame *if* a trace hook
  is installed

The proposed resolution to this problem is to take advantage of the fact that
whereas functions typically access their *own* namespace using the language
defined ``locals()`` builtin, trace functions necessarily use the implementation
dependent ``frame.f_locals`` interface, as a frame reference is what gets
passed to hook implementations.

Instead of being a direct reference to the internal frame value cache historically
returned by the ``locals()`` builtin, the Python level ``frame.f_locals`` will be
updated to instead return instances of a dedicated fast locals proxy type that
writes and reads values directly to and from the fast locals array on the
underlying frame. Each access of the attribute produces a new instance of the
proxy (so creating proxy instances is intentionally a cheap operation).

Despite the new proxy type becoming the preferred way to access local variables
on optimised frames, the internal value cache stored on the frame is still
retained for two key purposes:

* maintaining backwards compatibility for and interoperability with the
  ``PyEval_GetLocals()`` C API
* providing storage space for additional keys that don't have slots in the
  fast locals array (e.g. the ``__return__`` and ``__exception__`` keys set by
  ``pdb`` when tracing code execution for debugging purposes)


With the changes in this PEP, this internal frame value cache is no longer
directly accessible from Python code (whereas historically it was both
returned by the ``locals()`` builtin and available as the ``frame.f_locals``
attribute). Instead, the value cache is only accessible via the
``PyEval_GetLocals()`` C API and by directly accessing the internal storage of
a frame object.

Fast locals proxy objects and the internal frame value cache returned by
``PyEval_GetLocals()`` offer the following behavioural guarantees:

* changes made via a fast locals proxy will be immediately visible to the frame
  itself, to other fast locals proxy objects for the same frame, and in the
  internal value cache stored on the frame (it is this last point that provides
  ``PyEval_GetLocals()`` interoperability)
* changes made directly to the internal frame value cache will never be visible
  to the frame itself, and will only be reliably visible via fast locals proxies
  for the same frame if the change relates to extra variables that don't have
  slots in the frame's fast locals array
* changes made by executing code in the frame will be immediately visible to all
  fast locals proxy objects for that frame (both existing proxies and newly
  created ones). Visibility in the internal frame value cache cache returned
  by ``PyEval_GetLocals()`` is subject to the cache update guidelines discussed
  in the next section

As a result of these points, only code using ``PyEval_GetLocals()``,
``PyLocals_Get()``, or ``PyLocals_GetCopy()`` will need to be concerned about
the frame value cache potentially becoming stale. Code using the new frame fast
locals proxy API (whether from Python or from C) will always see the live state
of the frame.


Fast locals proxy implementation details
----------------------------------------

Each fast locals proxy instance has a single internal attribute that is not
exposed as part of the Python runtime API:

* *frame*: the underlying optimised frame that the proxy provides access to

In addition, proxy instances use and update the following attributes stored on the
underlying frame or code object:

* *_name_to_offset_mapping*: a hidden mapping from variable names to fast local
  storage offsets. This mapping is lazily initialized on the first frame read or
  write access through a fast locals proxy, rather than being eagerly populated
  as soon as the first fast locals proxy is created. Since the mapping is
  identical for all frames running a given code object, a single copy is stored
  on the code object, rather than each frame object populating its own mapping
* *locals*: the internal frame value cache returned by the ``PyEval_GetLocals()``
  C API and updated by the ``PyFrame_FastToLocals()`` C API. This is the mapping
  that the ``locals()`` builtin returns in Python 3.10 and earlier.

``__getitem__`` operations on the proxy will populate the ``_name_to_offset_mapping``
on the code object (if it is not already populated), and then either return the
relevant value (if the key is found in either the ``_name_to_offset_mapping``
mapping or the internal frame value cache), or else raise ``KeyError``. Variables
that are defined on the frame but not currently bound also raise ``KeyError``
(just as they're omitted from the result of ``locals()``).

As the frame storage is always accessed directly, the proxy will automatically
pick up name binding and unbinding operations that take place as the function
executes. The internal value cache is implicitly updated when individual
variables are read from the frame state (including for containment checks,
which need to check if the name is currently bound or unbound).

Similarly, ``__setitem__`` and ``__delitem__`` operations on the proxy will
directly affect the corresponding fast local or cell reference on the underlying
frame, ensuring that changes are immediately visible to the running Python code,
rather than needing to be written back to the runtime storage at some later time.
Such changes are also immediately written to the internal frame value cache to
make them visible to users of the ``PyEval_GetLocals()`` C API.

Keys that are not defined as local or closure variables on the underlying frame
are still written to the internal value cache on optimised frames. This allows
utilities like ``pdb`` (which writes ``__return__`` and ``__exception__``
values into the frame's ``f_locals`` mapping) to continue working as they always
have. These additional keys that do not correspond to a local or closure
variable on the frame will be left alone by future cache sync operations.
Using the frame value cache to store these extra keys (rather than defining a
new mapping that holds only the extra keys) provides full interoperability
with the existing ``PyEval_GetLocals()`` API (since users of either API will
see extra keys added by users of either API, rather than users of the new fast
locals proxy API only seeing keys added via that API).

An additional benefit of storing only the variable value cache on the frame
(rather than storing an instance of the proxy type), is that it avoids
creating a reference cycle from the frame back to itself, so the frame will
only be kept alive if another object retains a reference to a proxy instance.

Note: calling the ``proxy.clear()`` method has a similarly broad impact as
calling ``PyFrame_LocalsToFast()`` on an empty frame value cache in earlier
versions. Not only will the frame local variables be cleared, but also any cell
variables accessible from the frame (whether those cells are owned by the
frame itself or by an outer frame). This *can* clear a class's ``__class__``
cell if called on the frame of a method that uses the zero-arg ``super()``
construct (or otherwise references ``__class__``). This exceeds the scope of
calling ``frame.clear()``, as that only drop's the frame's references to cell
variables, it doesn't clear the cells themselves. This PEP could be a potential
opportunity to narrow the scope of attempts to clear the frame variables
directly by leaving cells belonging to outer frames alone, and only clearing
local variables and cells belonging directly to the frame underlying the proxy
(this issue affects :pep:`667` as well, as the question relates to the handling of
cell variables, and is entirely independent of the internal frame value cache).


Changes to the stable C API/ABI
-------------------------------

Unlike Python code, extension module functions that call in to the Python C API
can be called from any kind of Python scope. This means it isn't obvious from
the context whether ``locals()`` will return a snapshot or not, as it depends
on the scope of the calling Python code, not the C code itself.

This means it is desirable to offer C APIs that give predictable, scope
independent, behaviour. However, it is also desirable to allow C code to
exactly mimic the behaviour of Python code at the same scope.

To enable mimicking the behaviour of Python code, the stable C ABI would gain
the following new functions:

.. code-block:: c

    PyObject * PyLocals_Get();
    PyLocals_Kind PyLocals_GetKind();

``PyLocals_Get()`` is directly equivalent to the Python ``locals()`` builtin.
It returns a new reference to the local namespace mapping for the active
Python frame at module and class scope, and when using ``exec()`` or ``eval()``.
It returns a shallow copy of the active namespace at
function/coroutine/generator scope.

``PyLocals_GetKind()`` returns a value from the newly defined ``PyLocals_Kind``
enum, with the following options being available:

* ``PyLocals_DIRECT_REFERENCE``: ``PyLocals_Get()`` returns a direct reference
  to the local namespace for the running frame.
* ``PyLocals_SHALLOW_COPY``: ``PyLocals_Get()`` returns a shallow copy of the
  local namespace for the running frame.
* ``PyLocals_UNDEFINED``: an error occurred (e.g. no active Python thread
  state). A Python exception will be set if this value is returned.

Since the enum is used in the stable ABI, an additional 31-bit value is set to
ensure that it is safe to cast arbitrary signed 32-bit signed integers to
``PyLocals_Kind`` values.

This query API allows extension module code to determine the potential impact
of mutating the mapping returned by ``PyLocals_Get()`` without needing access
to the details of the running frame object. Python code gets equivalent
information visually through lexical scoping (as covered in the new ``locals()``
builtin documentation).

To allow extension module code to behave consistently regardless of the active
Python scope, the stable C ABI would gain the following new function:

.. code-block:: c

    PyObject * PyLocals_GetCopy();

``PyLocals_GetCopy()`` returns a new dict instance populated from the current
locals namespace. Roughly equivalent to ``dict(locals())`` in Python code, but
avoids the double-copy in the case where ``locals()`` already returns a shallow
copy. Akin to the following code, but doesn't assume there will only ever be
two kinds of locals result:

.. code-block:: c

    locals = PyLocals_Get();
    if (PyLocals_GetKind() == PyLocals_DIRECT_REFERENCE) {
        locals = PyDict_Copy(locals);
    }

The existing ``PyEval_GetLocals()`` API will retain its existing behaviour in
CPython (mutable locals at class and module scope, shared dynamic snapshot
otherwise). However, its documentation will be updated to note that the
conditions under which the shared dynamic snapshot get updated have changed.

The ``PyEval_GetLocals()`` documentation will also be updated to recommend
replacing usage of this API with whichever of the new APIs is most appropriate
for the use case:

* Use ``PyLocals_Get()`` (optionally combined with ``PyDictProxy_New()``) for
  read-only access to the current locals namespace. This form of usage will
  need to be aware that the copy may go stale in optimised frames.
* Use ``PyLocals_GetCopy()`` for a regular mutable dict that contains a copy of
  the current locals namespace, but has no ongoing connection to the active
  frame.
* Use ``PyLocals_Get()`` to exactly match the semantics of the Python level
  ``locals()`` builtin.
* Query ``PyLocals_GetKind()`` explicitly to implement custom handling
  (e.g. raising a meaningful exception) for scopes where ``PyLocals_Get()``
  would return a shallow copy rather than granting read/write access to the
  locals namespace.
* Use implementation specific APIs (e.g. ``PyObject_GetAttrString(frame, "f_locals")``)
  if read/write access to the frame is required and ``PyLocals_GetKind()``
  returns something other than ``PyLocals_DIRECT_REFERENCE``.


Changes to the public CPython C API
-----------------------------------

The existing ``PyEval_GetLocals()`` API returns a borrowed reference, which
means it cannot be updated to return the new shallow copies at function
scope. Instead, it will continue to return a borrowed reference to an internal
dynamic snapshot stored on the frame object. This shared mapping will behave
similarly to the existing shared mapping in Python 3.10 and earlier, but the exact
conditions under which it gets refreshed will be different. Specifically, it
will be updated only in the following circumstance:

* any call to ``PyEval_GetLocals()``, ``PyLocals_Get()``, ``PyLocals_GetCopy()``,
  or the Python ``locals()`` builtin while the frame is running
* any call to ``PyFrame_GetLocals()``, ``PyFrame_GetLocalsCopy()``,
  ``_PyFrame_BorrowLocals()``, ``PyFrame_FastToLocals()``, or
  ``PyFrame_FastToLocalsWithError()`` for the frame
* any operation on a fast locals proxy object that updates the shared
  mapping as part of its implementation. In the initial reference
  implementation, those operations are those that are intrinsically ``O(n)``
  operations (``len(flp)``, mapping comparison, ``flp.copy()`` and rendering as
  a string), as well as those that refresh the cache entries for individual keys.

Requesting a fast locals proxy will *not* implicitly update the shared dynamic
snapshot, and the CPython trace hook handling will no longer implicitly update
it either.

(Note: even though ``PyEval_GetLocals()`` is part of the stable C API/ABI, the
specifics of when the namespace it returns gets refreshed are still an
interpreter implementation detail)

The additions to the public CPython C API are the frame level enhancements
needed to support the stable C API/ABI updates:

.. code-block:: c

    PyLocals_Kind PyFrame_GetLocalsKind(frame);
    PyObject * PyFrame_GetLocals(frame);
    PyObject * PyFrame_GetLocalsCopy(frame);
    PyObject * _PyFrame_BorrowLocals(frame);


``PyFrame_GetLocalsKind(frame)`` is the underlying API for
``PyLocals_GetKind()``.

``PyFrame_GetLocals(frame)`` is the underlying API for ``PyLocals_Get()``.

``PyFrame_GetLocalsCopy(frame)`` is the underlying API for
``PyLocals_GetCopy()``.

``_PyFrame_BorrowLocals(frame)`` is the underlying API for
``PyEval_GetLocals()``. The underscore prefix is intended to discourage use and
to indicate that code using it is unlikely to be portable across
implementations. However, it is documented and visible to the linker in order
to avoid having to access the internals of the frame struct from the
``PyEval_GetLocals()`` implementation.

The ``PyFrame_LocalsToFast()`` function will be changed to always emit
``RuntimeError``, explaining that it is no longer a supported operation, and
affected code should be updated to use
``PyObject_GetAttrString(frame, "f_locals")`` to obtain a read/write proxy
instead.

In addition to the above documented interfaces, the draft reference
implementation also exposes the following undocumented interfaces:

.. code-block:: c

    PyTypeObject _PyFastLocalsProxy_Type;
    #define _PyFastLocalsProxy_CheckExact(self) Py_IS_TYPE(op, &_PyFastLocalsProxy_Type)

This type is what the reference implementation actually returns from
``PyObject_GetAttrString(frame, "f_locals")`` for optimized frames (i.e.
when ``PyFrame_GetLocalsKind()`` returns ``PyLocals_SHALLOW_COPY``).


Reducing the runtime overhead of trace hooks
--------------------------------------------

As noted in [9]_, the implicit call to ``PyFrame_FastToLocals()`` in the
Python trace hook support isn't free, and could be rendered unnecessary if
the frame proxy read values directly from the frame instead of getting them
from the mapping.

As the new frame locals proxy type doesn't require separate data refresh steps,
this PEP incorporates Victor Stinner's proposal to no longer implicitly call
``PyFrame_FastToLocalsWithError()`` before calling trace hooks implemented in
Python.

Code using the new fast locals proxy objects will have the dynamic locals snapshot
implicitly refreshed when accessing methods that need it, while code using the
``PyEval_GetLocals()`` API will implicitly refresh it when making that call.

The PEP necessarily also drops the implicit call to ``PyFrame_LocalsToFast()``
when returning from a trace hook, as that API now always raises an exception.


Rationale and Design Discussion
===============================

Changing ``locals()`` to return independent snapshots at function scope
-----------------------------------------------------------------------

The ``locals()`` builtin is a required part of the language, and in the
reference implementation it has historically returned a mutable mapping with
the following characteristics:

* each call to ``locals()`` returns the *same* mapping object
* for namespaces where ``locals()`` returns a reference to something other than
  the actual local execution namespace, each call to ``locals()`` updates the
  mapping object with the current state of the local variables and any referenced
  nonlocal cells
* changes to the returned mapping *usually* aren't written back to the
  local variable bindings or the nonlocal cell references, but write backs
  can be triggered by doing one of the following:

  * installing a Python level trace hook (write backs then happen whenever
    the trace hook is called)
  * running a function level wildcard import (requires bytecode injection in Py3)
  * running an ``exec`` statement in the function's scope (Py2 only, since
    ``exec`` became an ordinary builtin in Python 3)

Originally this PEP proposed to retain the first two of these properties,
while changing the third in order to address the outright behaviour bugs that
it can cause.

In [7]_ Nathaniel Smith made a persuasive case that we could make the behaviour
of ``locals()`` at function scope substantially less confusing by retaining only
the second property and having each call to ``locals()`` at function scope
return an *independent* snapshot of the local variables and closure references
rather than updating an implicitly shared snapshot.

As this revised design also made the implementation markedly easier to follow,
the PEP was updated to propose this change in behaviour, rather than retaining
the historical shared snapshot.


Keeping ``locals()`` as a snapshot at function scope
----------------------------------------------------

As discussed in [7]_, it would theoretically be possible to change the semantics
of the ``locals()`` builtin to return the write-through proxy at function scope,
rather than switching it to return independent snapshots.

This PEP doesn't (and won't) propose this as it's a backwards incompatible
change in practice, even though code that relies on the current behaviour is
technically operating in an undefined area of the language specification.

Consider the following code snippet::

    def example():
        x = 1
        locals()["x"] = 2
        print(x)

Even with a trace hook installed, that function will consistently print ``1``
on the current reference interpreter implementation::

    >>> example()
    1
    >>> import sys
    >>> def basic_hook(*args):
    ...     return basic_hook
    ...
    >>> sys.settrace(basic_hook)
    >>> example()
    1

Similarly, ``locals()`` can be passed to the ``exec()`` and ``eval()`` builtins
at function scope (either explicitly or implicitly) without risking unexpected
rebinding of local variables or closure references.

Provoking the reference interpreter into incorrectly mutating the local variable
state requires a more complex setup where a nested function closes over a
variable being rebound in the outer function, and due to the use of either
threads, generators, or coroutines, it's possible for a trace function to start
running for the nested function before the rebinding operation in the outer
function, but finish running after the rebinding operation has taken place (in
which case the rebinding will be reverted, which is the bug reported in [1]_).

In addition to preserving the de facto semantics which have been in place since
:pep:`227` introduced nested scopes in Python 2.1, the other benefit of restricting
the write-through proxy support to the implementation-defined frame object API
is that it means that only interpreter implementations which emulate the full
frame API need to offer the write-through capability at all, and that
JIT-compiled implementations only need to enable it when a frame introspection
API is invoked, or a trace hook is installed, not whenever ``locals()`` is
accessed at function scope.

Returning snapshots from ``locals()`` at function scope also means that static
analysis for function level code will be more reliable, as only access to the
frame machinery will allow rebinding of local and nonlocal variable
references in a way that is hidden from static analysis.


What happens with the default args for ``eval()`` and ``exec()``?
-----------------------------------------------------------------

These are formally defined as inheriting ``globals()`` and ``locals()`` from
the calling scope by default.

There isn't any need for the PEP to change these defaults, so it doesn't, and
``exec()`` and ``eval()`` will start running in a shallow copy of the local
namespace when that is what ``locals()`` returns.

This behaviour will have potential performance implications, especially
for functions with large numbers of local variables (e.g. if these functions
are called in a loop, calling ``globals()`` and ``locals()`` once before the
loop and then passing the namespace into the function explicitly will give the
same semantics and performance characteristics as the status quo, whereas
relying on the implicit default would create a new shallow copy of the local
namespace on each iteration).

(Note: the reference implementation draft PR has updated the ``locals()`` and
``vars()``, ``eval()``, and ``exec()`` builtins to use ``PyLocals_Get()``. The
``dir()`` builtin still uses ``PyEval_GetLocals()``, since it's only using it
to make a list from the keys).


.. _pep-558-exec-eval-impact:

Additional considerations for ``eval()`` and ``exec()`` in optimized scopes
---------------------------------------------------------------------------

Note: while implementing :pep:`667`, it was noted that neither that PEP nor this one
clearly explained the impact the ``locals()`` changes would have on code execution APIs
like ``exec()`` and ``eval()``. This section was added to this PEP's rationale to better
describe the impact and explain the intended benefits of the change.

When ``exec()`` was converted from a statement to a builtin function
in Python 3.0 (part of the core language changes in :pep:`3100`), the
associated implicit call to ``PyFrame_LocalsToFast()`` was removed, so
it typically appears as if attempts to write to local variables with
``exec()`` in optimized frames are ignored::

    >>> def f():
    ...     x = 0
    ...     exec("x = 1")
    ...     print(x)
    ...     print(locals()["x"])
    ...
    >>> f()
    0
    0

In truth, the writes aren't being ignored, they just aren't
being copied from the dictionary cache back to the optimized local
variable array. The changes to the dictionary are then overwritten
the next time the dictionary cache is refreshed from the array::

    >>> def f():
    ...     x = 0
    ...     locals_cache = locals()
    ...     exec("x = 1")
    ...     print(x)
    ...     print(locals_cache["x"])
    ...     print(locals()["x"])
    ...
    >>> f()
    0
    1
    0

.. _pep-558-ctypes-example:

The behaviour becomes even stranger if a tracing function
or another piece of code invokes ``PyFrame_LocalsToFast()`` before
the cache is next refreshed. In those cases the change *is*
written back to the optimized local variable array::

    >>> from sys import _getframe
    >>> from ctypes import pythonapi, py_object, c_int
    >>> _locals_to_fast = pythonapi.PyFrame_LocalsToFast
    >>> _locals_to_fast.argtypes = [py_object, c_int]
    >>> def f():
    ...     _frame = _getframe()
    ...     _f_locals = _frame.f_locals
    ...     x = 0
    ...     exec("x = 1")
    ...     _locals_to_fast(_frame, 0)
    ...     print(x)
    ...     print(locals()["x"])
    ...     print(_f_locals["x"])
    ...
    >>> f()
    1
    1
    1

This situation was more common in Python 3.10 and earlier
versions, as merely installing a tracing function was enough
to trigger implicit calls to ``PyFrame_LocalsToFast()`` after
every line of Python code. However, it can still happen in Python
3.11+ depending on exactly which tracing functions are active
(e.g. interactive debuggers intentionally do this so that changes
made at the debugging prompt are visible when code execution
resumes).

All of the above comments in relation to ``exec()`` apply to
*any* attempt to mutate the result of ``locals()`` in optimized
scopes, and are the main reason that the ``locals()`` builtin
docs contain this caveat:

    Note: The contents of this dictionary should not be modified;
    changes may not affect the values of local and free variables
    used by the interpreter.

While the exact wording in the library reference is not entirely explicit,
both ``exec()`` and ``eval()`` have long used the results of calling
``globals()`` and ``locals()`` in the calling Python frame as their default
execution namespace.

This was historically also equivalent to using the calling frame's
``frame.f_globals`` and ``frame.f_locals`` attributes, but this PEP maps
the default namespace arguments for ``exec()`` and ``eval()`` to
``globals()`` and ``locals()`` in the calling frame in order to preserve
the property of defaulting to ignoring attempted writes to the local
namespace in optimized scopes.

This poses a potential compatibility issue for some code, as with the
previous implementation that returns the same dict when ``locals()`` is called
multiple times in function scope, the following code usually worked due to
the implicitly shared local variable namespace::

    def f():
        exec('a = 0')  # equivalent to exec('a = 0', globals(), locals())
        exec('print(a)')  # equivalent to exec('print(a)', globals(), locals())
        print(locals())  # {'a': 0}
        # However, print(a) will not work here
    f()

With ``locals()`` in an optimised scope returning the same shared dict for each call,
it was possible to store extra "fake locals" in that dict. While these aren't real
locals known by the compiler (so they can't be printed with code like ``print(a)``),
they can still be accessed via ``locals()`` and shared between multiple ``exec()``
calls in the same function scope. Furthermore, because they're *not* real locals,
they don't get implicitly updated or removed when the shared cache is refreshed
from the local variable storage array.

When the code in ``exec()`` tries to write to an existing local variable, the
runtime behaviour gets harder to predict::

    def f():
        a = None
        exec('a = 0')  # equivalent to exec('a = 0', globals(), locals())
        exec('print(a)')  # equivalent to exec('print(a)', globals(), locals())
        print(locals())  # {'a': None}
    f()

``print(a)`` will print ``None`` because the implicit ``locals()`` call in
``exec()`` refreshes the cached dict with the actual values on the frame.
This means that, unlike the "fake" locals created by writing back to ``locals()``
(including via previous calls to ``exec()``), the real locals known by the
compiler can't easily be modified by ``exec()`` (it can be done, but it requires
both retrieving the ``frame.f_locals`` attribute to enable writes back to the frame,
and then invoking ``PyFrame_LocalsToFast()``, as :ref:`shown <pep-558-ctypes-example>`
using ``ctypes`` above).

As noted in the :ref:`pep-558-motivation` section, this confusing side effect
happens even if the local variable is only defined *after* the ``exec()`` calls::

    >>> def f():
    ...     exec("a = 0")
    ...     exec("print('a' in locals())") # Printing 'a' directly won't work
    ...     print(locals())
    ...     a = None
    ...     print(locals())
    ...
    >>> f()
    False
    {}
    {'a': None}

Because ``a`` is a real local variable that is not currently bound to a value, it
gets explicitly removed from the dictionary returned by ``locals()`` whenever
``locals()`` is called prior to the ``a = None`` line. This removal is intentional,
as it allows the contents of ``locals()`` to be updated correctly in optimized
scopes when ``del`` statements are used to delete previously bound local variables.

As noted in the ``ctypes`` :ref:`example <pep-558-ctypes-example>`, the above behavioural
description may be invalidated if the CPython ``PyFrame_LocalsToFast()`` API gets invoked
while the frame is still running. In that case, the changes to ``a`` *might* become visible
to the running code, depending on exactly when that API is called (and whether the frame
has been primed for locals modification by accessing the ``frame.f_locals`` attribute).

As described above, two options were considered to replace this confusing behaviour:

* make ``locals()`` return write-through proxy instances (similar
  to ``frame.f_locals``)
* make ``locals()`` return genuinely independent snapshots so that
  attempts to change the values of local variables via ``exec()``
  would be *consistently* ignored without any of the caveats
  noted above.

The PEP chooses the second option for the following reasons:

* returning independent snapshots in optimized scopes preserves
  the Python 3.0 change to ``exec()`` that resulted in attempts
  to mutate local variables via ``exec()`` being ignored in most
  cases
* the distinction between "``locals()`` gives an instantaneous
  snapshot of the local variables in optimized scopes, and
  read/write access in other scopes" and "``frame.f_locals``
  gives read/write access to the local variables in all scopes,
  including optimized scopes" allows the intent of a piece of
  code to be clearer than it would be if both APIs granted
  full read/write access in optimized scopes, even when write
  access wasn't needed or desired
* in addition to improving clarity for human readers, ensuring
  that name rebinding in optimized scopes remains lexically
  visible in the code (as long as the frame introspection APIs
  are not accessed) allows compilers and interpreters to apply
  related performance optimizations more consistently
* only Python implementations that support the optional frame
  introspection APIs will need to provide the new write-through
  proxy support for optimized frames

With the semantic changes to ``locals()`` in this PEP, it becomes much easier to explain
the behavior of ``exec()`` and ``eval()``: in optimized scopes, they will *never* implicitly
affect local variables; in other scopes, they will *always* implicitly affect local
variables. In optimized scopes, any implicit assignment to the local variables will be
discarded when the code execution API returns, since a fresh copy of the local variables
is used on each invocation.


Retaining the internal frame value cache
----------------------------------------

Retaining the internal frame value cache results in some visible quirks when
frame proxy instances are kept around and re-used after name binding and
unbinding operations have been executed on the frame.

The primary reason for retaining the frame value cache is to maintain backwards
compatibility with the ``PyEval_GetLocals()`` API. That API returns a borrowed
reference, so it must refer to persistent state stored on the frame object.
Storing a fast locals proxy object on the frame creates a problematic reference
cycle, so the cleanest option is to instead continue to return a frame value
cache, just as this function has done since optimised frames were first
introduced.

With the frame value cache being kept around anyway, it then further made sense
to rely on it to simplify the fast locals proxy mapping implementation.

Note: the fact :pep:`667` *doesn't* use the internal frame value cache as part of the
write-through proxy implementation is the key Python level difference between the two PEPs.


Changing the frame API semantics in regular operation
-----------------------------------------------------

Note: when this PEP was first written, it predated the Python 3.11 change to drop the
implicit writeback of the frame local variables whenever a tracing function was installed,
so making that change was included as part of the proposal.

Earlier versions of this PEP proposed having the semantics of the frame
``f_locals`` attribute depend on whether or not a tracing hook was currently
installed - only providing the write-through proxy behaviour when a tracing hook
was active, and otherwise behaving the same as the historical ``locals()``
builtin.

That was adopted as the original design proposal for a couple of key reasons,
one pragmatic and one more philosophical:

* Object allocations and method wrappers aren't free, and tracing functions
  aren't the only operations that access frame locals from outside the function.
  Restricting the changes to tracing mode meant that the additional memory and
  execution time overhead of these changes would be as close to zero in regular
  operation as we can possibly make them.
* "Don't change what isn't broken": the current tracing mode problems are caused
  by a requirement that's specific to tracing mode (support for external
  rebinding of function local variable references), so it made sense to also
  restrict any related fixes to tracing mode

However, actually attempting to implement and document that dynamic approach
highlighted the fact that it makes for a really subtle runtime state dependent
behaviour distinction in how ``frame.f_locals`` works, and creates several
new edge cases around how ``f_locals`` behaves as trace functions are added
and removed.

Accordingly, the design was switched to the current one, where
``frame.f_locals`` is always a write-through proxy, and ``locals()`` is always
a snapshot, which is both simpler to implement and easier to explain.

Regardless of how the CPython reference implementation chooses to handle this,
optimising compilers and interpreters also remain free to impose additional
restrictions on debuggers, such as making local variable mutation through frame
objects an opt-in behaviour that may disable some optimisations (just as the
emulation of CPython's frame API is already an opt-in flag in some Python
implementations).


Continuing to support storing additional data on optimised frames
-----------------------------------------------------------------

One of the draft iterations of this PEP proposed removing the ability to store
additional data on optimised frames by writing to ``frame.f_locals`` keys that
didn't correspond to local or closure variable names on the underlying frame.

While this idea offered some attractive simplification of the fast locals proxy
implementation, ``pdb`` stores ``__return__`` and ``__exception__`` values on
arbitrary frames, so the standard library test suite fails if that functionality
no longer works.

Accordingly, the ability to store arbitrary keys was retained, at the expense
of certain operations on proxy objects being slower than could otherwise be
(since they can't assume that only names defined on the code object will be
accessible through the proxy).

It is expected that the exact details of the interaction between the fast locals
proxy and the ``f_locals`` value cache on the underlying frame will evolve over
time as opportunities for improvement are identified.


Historical semantics at function scope
--------------------------------------

The current semantics of mutating ``locals()`` and ``frame.f_locals`` in CPython
are rather quirky due to historical implementation details:

* actual execution uses the fast locals array for local variable bindings and
  cell references for nonlocal variables
* there's a ``PyFrame_FastToLocals`` operation that populates the frame's
  ``f_locals`` attribute based on the current state of the fast locals array
  and any referenced cells. This exists for three reasons:

  * allowing trace functions to read the state of local variables
  * allowing traceback processors to read the state of local variables
  * allowing ``locals()`` to read the state of local variables
* a direct reference to ``frame.f_locals`` is returned from ``locals()``, so if
  you hand out multiple concurrent references, then all those references will be
  to the exact same dictionary
* the two common calls to the reverse operation, ``PyFrame_LocalsToFast``, were
  removed in the migration to Python 3: ``exec`` is no longer a statement (and
  hence can no longer affect function local namespaces), and the compiler now
  disallows the use of ``from module import *`` operations at function scope
* however, two obscure calling paths remain: ``PyFrame_LocalsToFast`` is called
  as part of returning from a trace function (which allows debuggers to make
  changes to the local variable state), and you can also still inject the
  ``IMPORT_STAR`` opcode when creating a function directly from a code object
  rather than via the compiler

This proposal deliberately *doesn't* formalise these semantics as is, since they
only make sense in terms of the historical evolution of the language and the
reference implementation, rather than being deliberately designed.


Proposing several additions to the stable C API/ABI
---------------------------------------------------

Historically, the CPython C API (and subsequently, the stable ABI) has
exposed only a single API function related to the Python ``locals`` builtin:
``PyEval_GetLocals()``. However, as it returns a borrowed reference, it is
not possible to adapt that interface directly to supporting the new ``locals()``
semantics proposed in this PEP.

An earlier iteration of this PEP proposed a minimalist adaptation to the new
semantics: one C API function that behaved like the Python ``locals()`` builtin,
and another that behaved like the ``frame.f_locals`` descriptor (creating and
returning the write-through proxy if necessary).

The feedback [8]_ on that version of the C API was that it was too heavily based
on how the Python level semantics were implemented, and didn't account for the
behaviours that authors of C extensions were likely to *need*.

The broader API now being proposed came from grouping the potential reasons for
wanting to access the Python ``locals()`` namespace from an extension module
into the following cases:

* needing to exactly replicate the semantics of the Python level ``locals()``
  operation. This is the ``PyLocals_Get()`` API.
* needing to behave differently depending on whether writes to the result of
  ``PyLocals_Get()`` will be visible to Python code or not. This is handled by
  the ``PyLocals_GetKind()`` query API.
* always wanting a mutable namespace that has been pre-populated from the
  current Python ``locals()`` namespace, but *not* wanting any changes to
  be visible to Python code. This is the ``PyLocals_GetCopy()`` API.
* always wanting a read-only view of the current locals namespace, without
  incurring the runtime overhead of making a full copy each time. This isn't
  readily offered for optimised frames due to the need to check whether names
  are currently bound or not, so no specific API is being added to cover it.

Historically, these kinds of checks and operations would only have been
possible if a Python implementation emulated the full CPython frame API. With
the proposed API, extension modules can instead ask more clearly for the
semantics that they actually need, giving Python implementations more
flexibility in how they provide those capabilities.


Comparison with PEP 667
-----------------------

NOTE: the comparison below is against PEP 667 as it was in December 2021.
It does not reflect the state of PEP 667 as of April 2024 (when this PEP was
withdrawn in favour of proceeding with PEP 667).

:pep:`667` offers a partially competing proposal for this PEP that suggests it
would be reasonable to eliminate the internal frame value cache on optimised
frames entirely.

These changes were originally offered as amendments to :pep:`558`, and the PEP
author rejected them for three main reasons:

* the initial claim that ``PyEval_GetLocals()`` was unfixable because it returns
  a borrowed reference was simply false, as it is still working in the :pep:`558`
  reference implementation. All that is required to keep it working is to
  retain the internal frame value cache and design the fast locals proxy in
  such a way that it is reasonably straightforward to keep the cache up to date
  with changes in the frame state without incurring significant runtime overhead
  when the cache isn't needed. Given that this claim is false, the proposal to
  require that all code using the ``PyEval_GetLocals()`` API be rewritten to use
  a new API with different refcounting semantics fails :pep:`387`'s requirement
  that API compatibility breaks should have a large benefit to breakage ratio
  (since there's no significant benefit gained from dropping the cache, no code
  breakage can be justified). The only genuinely unfixable public API is
  ``PyFrame_LocalsToFast()`` (which is why both PEPs propose breaking that).
* without some form of internal value cache, the API performance characteristics
  of the fast locals proxy mapping become quite unintuitive. ``len(proxy)``, for
  example, becomes consistently O(n) in the number of variables defined on the
  frame, as the proxy has to iterate over the entire fast locals array to see
  which names are currently bound to values before it can determine the answer.
  By contrast, maintaining an internal frame value cache potentially allows
  proxies to largely be treated as normal dictionaries from an algorithmic
  complexity point of view, with allowances only needing to be made for the
  initial implicit O(n) cache refresh that runs the first time an operation
  that relies on the cache being up to date is executed.
* the claim that a cache-free implementation would be simpler is highly suspect,
  as :pep:`667` includes only a pure Python sketch of a subset of a mutable mapping
  implementation, rather than a full-fledged C implementation of a new mapping
  type integrated with the underlying data storage for optimised frames.
  :pep:`558`'s fast locals proxy implementation delegates heavily to the
  frame value cache for the operations needed to fully implement the mutable
  mapping API, allowing it to re-use the existing dict implementations of the
  following operations:

  * ``__len__``
  * ``__str__``
  * ``__or__`` (dict union)
  * ``__iter__`` (allowing the ``dict_keyiterator`` type to be reused)
  * ``__reversed__`` (allowing the ``dict_reversekeyiterator`` type to be reused)
  * ``keys()`` (allowing the ``dict_keys`` type to be reused)
  * ``values()`` (allowing the ``dict_values`` type to be reused)
  * ``items()`` (allowing the ``dict_items`` type to be reused)
  * ``copy()``
  * ``popitem()``
  * value comparison operations

Of the three reasons, the first is the most important (since we need compelling
reasons to break API backwards compatibility, and we don't have them).

However, after reviewing :pep:`667`'s proposed Python level semantics, the author
of this PEP eventually agreed that they *would* be simpler for users of the
Python ``locals()`` API, so this distinction between the two PEPs has been
eliminated: regardless of which PEP and implementation is accepted, the fast
locals proxy object *always* provides a consistent view of the current state
of the local variables, even if this results in some operations becoming O(n)
that would be O(1) on a regular dictionary (specifically, ``len(proxy)``
becomes O(n), since it needs to check which names are currently bound, and proxy
mapping comparisons avoid relying on the length check optimisation that allows
differences in the number of stored keys to be detected quickly for regular
mappings).

Due to the adoption of these non-standard performance characteristics in the
proxy implementation, the ``PyLocals_GetView()`` and ``PyFrame_GetLocalsView()``
C APIs were also removed from the proposal in this PEP.

This leaves the only remaining points of distinction between the two PEPs as
specifically related to the C API:

* :pep:`667` still proposes completely unnecessary C API breakage (the programmatic
  deprecation and eventual removal of ``PyEval_GetLocals()``,
  ``PyFrame_FastToLocalsWithError()``, and ``PyFrame_FastToLocals()``) without
  justification, when it is entirely possible to keep these working indefinitely
  (and interoperably) given a suitably designed fast locals proxy implementation
* the fast locals proxy handling of additional variables is defined in this PEP
  in a way that is fully interoperable with the existing ``PyEval_GetLocals()``
  API. In the proxy implementation proposed in :pep:`667`, users of the new frame
  API will not see changes made to additional variables by users of the old API,
  and changes made to additional variables via the old API will be overwritten
  on subsequent calls to ``PyEval_GetLocals()``.
* the ``PyLocals_Get()`` API in this PEP is called ``PyEval_Locals()`` in :pep:`667`.
  This function name is a bit strange as it lacks a verb, making it look more
  like a type name than a data access API.
* this PEP adds ``PyLocals_GetCopy()`` and ``PyFrame_GetLocalsCopy()`` APIs to
  allow extension modules to easily avoid incurring a double copy operation in
  frames where ``PyLocals_Get()`` already makes a copy
* this PEP adds ``PyLocals_Kind``, ``PyLocals_GetKind()``, and
  ``PyFrame_GetLocalsKind()`` to allow extension modules to identify when code
  is running at function scope without having to inspect non-portable frame and
  code object APIs (without the proposed query API, the existing equivalent to
  the new ``PyLocals_GetKind() == PyLocals_SHALLOW_COPY`` check is to include
  the CPython internal frame API headers and check if
  ``_PyFrame_GetCode(PyEval_GetFrame())->co_flags & CO_OPTIMIZED`` is set)

The Python pseudo-code below is based on the implementation sketch presented
in :pep:`667` as of the time of writing (2021-10-24). The differences that
provide the improved interoperability between the new fast locals proxy API
and the existing ``PyEval_GetLocals()`` API are noted in comments.

As in :pep:`667`, all attributes that start with an underscore are invisible and
cannot be accessed directly. They serve only to illustrate the proposed design.

For simplicity (and as in :pep:`667`), the handling of module and class level
frames is omitted (they're much simpler, as ``_locals`` *is* the execution
namespace, so no translation is required).

::

    NULL: Object # NULL is a singleton representing the absence of a value.

    class CodeType:

        _name_to_offset_mapping_impl: dict | NULL
        ...

        def __init__(self, ...):
            self._name_to_offset_mapping_impl = NULL
            self._variable_names = deduplicate(
                self.co_varnames + self.co_cellvars + self.co_freevars
            )
            ...

        def _is_cell(self, offset):
            ... # How the interpreter identifies cells is an implementation detail

        @property
        def _name_to_offset_mapping(self):
            "Mapping of names to offsets in local variable array."
            if self._name_to_offset_mapping_impl is NULL:

                self._name_to_offset_mapping_impl = {
                    name: index for (index, name) in enumerate(self._variable_names)
                }
            return self._name_to_offset_mapping_impl

    class FrameType:

        _fast_locals : array[Object] # The values of the local variables, items may be NULL.
        _locals: dict | NULL # Dictionary returned by PyEval_GetLocals()

        def __init__(self, ...):
            self._locals = NULL
            ...

        @property
        def f_locals(self):
            return FastLocalsProxy(self)

    class FastLocalsProxy:

        __slots__ "_frame"

        def __init__(self, frame:FrameType):
            self._frame = frame

        def _set_locals_entry(self, name, val):
            f = self._frame
            if f._locals is NULL:
                f._locals = {}
            f._locals[name] = val

        def __getitem__(self, name):
            f = self._frame
            co = f.f_code
            if name in co._name_to_offset_mapping:
                index = co._name_to_offset_mapping[name]
                val = f._fast_locals[index]
                if val is NULL:
                    raise KeyError(name)
                if co._is_cell(offset)
                    val = val.cell_contents
                    if val is NULL:
                        raise KeyError(name)
                # PyEval_GetLocals() interop: implicit frame cache refresh
                self._set_locals_entry(name, val)
                return val
            # PyEval_GetLocals() interop: frame cache may contain additional names
            if f._locals is NULL:
                raise KeyError(name)
            return f._locals[name]

        def __setitem__(self, name, value):
            f = self._frame
            co = f.f_code
            if name in co._name_to_offset_mapping:
                index = co._name_to_offset_mapping[name]
                kind = co._local_kinds[index]
                if co._is_cell(offset)
                    cell = f._locals[index]
                    cell.cell_contents = val
                else:
                    f._fast_locals[index] = val
            # PyEval_GetLocals() interop: implicit frame cache update
            # even for names that are part of the fast locals array
            self._set_locals_entry(name, val)

        def __delitem__(self, name):
            f = self._frame
            co = f.f_code
            if name in co._name_to_offset_mapping:
                index = co._name_to_offset_mapping[name]
                kind = co._local_kinds[index]
                if co._is_cell(offset)
                    cell = f._locals[index]
                    cell.cell_contents = NULL
                else:
                    f._fast_locals[index] = NULL
            # PyEval_GetLocals() interop: implicit frame cache update
            # even for names that are part of the fast locals array
            if f._locals is not NULL:
                del f._locals[name]

        def __iter__(self):
            f = self._frame
            co = f.f_code
            for index, name in enumerate(co._variable_names):
                val = f._fast_locals[index]
                if val is NULL:
                    continue
                if co._is_cell(offset):
                    val = val.cell_contents
                    if val is NULL:
                        continue
                yield name
            for name in f._locals:
                # Yield any extra names not defined on the frame
                if name in co._name_to_offset_mapping:
                    continue
                yield name

        def popitem(self):
            f = self._frame
            co = f.f_code
            for name in self:
                val = self[name]
            # PyEval_GetLocals() interop: implicit frame cache update
            # even for names that are part of the fast locals array
            del name
            return name, val

        def _sync_frame_cache(self):
            # This method underpins PyEval_GetLocals, PyFrame_FastToLocals
            # PyFrame_GetLocals, PyLocals_Get, mapping comparison, etc
            f = self._frame
            co = f.f_code
            res = 0
            if f._locals is NULL:
                f._locals = {}
            for index, name in enumerate(co._variable_names):
                val = f._fast_locals[index]
                if val is NULL:
                    f._locals.pop(name, None)
                    continue
                if co._is_cell(offset):
                    if val.cell_contents is NULL:
                        f._locals.pop(name, None)
                        continue
                f._locals[name] = val

        def __len__(self):
            self._sync_frame_cache()
            return len(self._locals)

Note: the simplest way to convert the earlier iterations of the :pep:`558`
reference implementation into a preliminary implementation of the now proposed
semantics is to remove the ``frame_cache_updated`` checks in affected operations,
and instead always sync the frame cache in those methods. Adopting that approach
changes the algorithmic complexity of the following operations as shown (where
``n`` is the number of local and cell variables defined on the frame):

  * ``__len__``: O(1) -> O(n)
  * value comparison operations: no longer benefit from O(1) length check shortcut
  * ``__iter__``: O(1) -> O(n)
  * ``__reversed__``: O(1) -> O(n)
  * ``keys()``: O(1) -> O(n)
  * ``values()``: O(1) -> O(n)
  * ``items()``: O(1) -> O(n)
  * ``popitem()``: O(1) -> O(n)

The length check and value comparison operations have relatively limited
opportunities for improvement: without allowing usage of a potentially stale
cache, the only way to know how many variables are currently bound is to iterate
over all of them and check, and if the implementation is going to be spending
that many cycles on an operation anyway, it may as well spend it updating the
frame value cache and then consuming the result. These operations are O(n) in
both this PEP and in :pep:`667`. Customised implementations could be provided that
*are* faster than updating the frame cache, but it's far from clear that the
extra code complexity needed to speed these operations up would be worthwhile
when it only offers a linear performance improvement rather than an algorithmic
complexity improvement.

The O(1) nature of the other operations can be restored by adding implementation
code that doesn't rely on the value cache being up to date.

Keeping the iterator/iterable retrieval methods as O(1) will involve
writing custom replacements for the corresponding builtin dict helper types,
just as proposed in :pep:`667`. As illustrated above, the implementations would
be similar to the pseudo-code presented in :pep:`667`, but not identical (due to
the improved ``PyEval_GetLocals()`` interoperability offered by this PEP
affecting the way it stores extra variables).

``popitem()`` can be improved from "always O(n)" to "O(n) worst case" by
creating a custom implementation that relies on the improved iteration APIs.

To ensure stale frame information is never presented in the Python fast locals
proxy API, these changes in the reference implementation will need to be
implemented before merging.

The current implementation at time of writing (2021-10-24) also still stores a
copy of the fast refs mapping on each frame rather than storing a single
instance on the underlying code object (as it still stores cell references
directly, rather than check for cells on each fast locals array access). Fixing
this would also be required before merging.


Implementation
==============

The reference implementation update is in development as a draft pull
request on GitHub ([6]_).


Acknowledgements
================

Thanks to Nathaniel J. Smith for proposing the write-through proxy idea in
[1]_ and pointing out some critical design flaws in earlier iterations of the
PEP that attempted to avoid introducing such a proxy.

Thanks to Steve Dower and Petr Viktorin for asking that more attention be paid
to the developer experience of the proposed C API additions [8]_ [13]_.

Thanks to Larry Hastings for the suggestion on how to use enums in the stable
ABI while ensuring that they safely support typecasting from arbitrary
integers.

Thanks to Mark Shannon for pushing for further simplification of the C level
API and semantics, as well as significant clarification of the PEP text (and for
restarting discussion on the PEP in early 2021 after a further year of
inactivity) [10]_ [11]_ [12]_. Mark's comments that were ultimately published as
:pep:`667` also directly resulted in several implementation efficiency improvements
that avoid incurring the cost of redundant O(n) mapping refresh operations
when the relevant mappings aren't used, as well as the change to ensure that
the state reported through the Python level ``f_locals`` API is never stale.


References
==========

.. [1] `Broken local variable assignment given threads + trace hook + closure
   <https://github.com/python/cpython/issues/74929>`_

.. [3] `Updating function local variables from pdb is unreliable
   <https://github.com/python/cpython/issues/5384)>`_

.. [4] `CPython's Python API for installing trace hooks
   <https://docs.python.org/dev/library/sys.html#sys.settrace>`_

.. [5] `CPython's C API for installing trace hooks
   <https://docs.python.org/3/c-api/init.html#c.PyEval_SetTrace>`_

.. [6] `PEP 558 reference implementation
   <https://github.com/python/cpython/pull/3640/files>`_

.. [7] `Nathaniel's review of possible function level semantics for locals()
   <https://mail.python.org/pipermail/python-dev/2019-May/157738.html>`_

.. [8] `Discussion of more intentionally designed C API enhancements
   <https://discuss.python.org/t/pep-558-defined-semantics-for-locals/2936/3>`_

.. [9] `Disable automatic update of frame locals during tracing
   <https://github.com/python/cpython/issues/86363>`_

.. [10] `python-dev thread: Resurrecting PEP 558 (Defined semantics for locals())
   <https://mail.python.org/archives/list/python-dev@python.org/thread/TUQOEWQSCQZPUDV2UFFKQ3C3I4WGFPAJ/>`_

.. [11] `python-dev thread: Comments on PEP 558
   <https://mail.python.org/archives/list/python-dev@python.org/thread/A3UN4DGBCOB45STE6AQBITJFW6UZE43O/>`_

.. [12] `python-dev thread: More comments on PEP 558
   <https://mail.python.org/archives/list/python-dev@python.org/thread/7TKPMD5LHCBXGFUIMKDAUZELRH6EX76S/>`_

.. [13] `Petr Viktorin's suggestion to use an enum for PyLocals_Get's behaviour
   <https://mail.python.org/archives/list/python-dev@python.org/message/BTQUBHIVE766RPIWLORC5ZYRCRC4CEBL/>`_

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.