PEP: 667 Title: Consistent views of namespaces Author: Mark Shannon , Tian Gao Discussions-To: https://discuss.python.org/t/46631 Status: Accepted Type: Standards Track Created: 30-Jul-2021 Python-Version: 3.13 Post-History: 20-Aug-2021 Resolution: https://discuss.python.org/t/pep-667-consistent-views-of-namespaces/46631/25 Abstract ======== In early versions of Python all namespaces, whether in functions, classes or modules, were all implemented the same way: as a dictionary. For performance reasons, the implementation of function namespaces was changed. Unfortunately this meant that accessing these namespaces through ``locals()`` and ``frame.f_locals`` ceased to be consistent and some odd bugs crept in over the years as threads, generators and coroutines were added. This PEP proposes making these namespaces consistent once more. Modifications to ``frame.f_locals`` will always be visible in the underlying variables. Modifications to local variables will immediately be visible in ``frame.f_locals``, and they will be consistent regardless of threading or coroutines. The ``locals()`` function will act the same as it does now for class and modules scopes. For function scopes it will return an instantaneous snapshot of the underlying ``frame.f_locals``. Motivation ========== The current implementation of ``locals()`` and ``frame.f_locals`` is slow, inconsistent and buggy. We want to make it faster, consistent, and most importantly fix the bugs. For example:: class C: x = 1 sys._getframe().f_locals['x'] = 2 print(x) prints ``2`` but:: def f(): x = 1 sys._getframe().f_locals['x'] = 2 print(x) f() prints ``1`` This is inconsistent, and confusing. With this PEP both examples would print ``2``. Worse than that, the current behavior can result in strange `bugs `__. There are no compensating advantages for the current behavior; it is unreliable and slow. Rationale ========= The current implementation of ``frame.f_locals`` returns a dictionary that is created on the fly from the array of local variables. This can result in the array and dictionary getting out of sync with each other. Writes to the ``f_locals`` may not show up as modifications to local variables. Writes to local variables can get lost. By making ``frame.f_locals`` return a view on the underlying frame, these problems go away. ``frame.f_locals`` is always in sync with the frame because it is a view of it, not a copy of it. Specification ============= Python ------ ``frame.f_locals`` will return a view object on the frame that implements the ``collections.abc.Mapping`` interface. For module and class scopes ``frame.f_locals`` will be a dictionary, for function scopes it will be a custom class. ``locals()`` will be defined as:: def locals(): frame = sys._getframe(1) f_locals = frame.f_locals if frame.is_function(): f_locals = dict(f_locals) return f_locals All writes to the ``f_locals`` mapping will be immediately visible in the underlying variables. All changes to the underlying variables will be immediately visible in the mapping. The ``f_locals`` object will be a full mapping, and can have arbitrary key-value pairs added to it. For example:: def l(): "Get the locals of caller" return sys._getframe(1).f_locals def test(): if 0: y = 1 # Make 'y' a local variable x = 1 l()['x'] = 2 l()['y'] = 4 l()['z'] = 5 y print(locals(), x) ``test()`` will print ``{'x': 2, 'y': 4, 'z': 5} 2``. In Python 3.10, the above will fail with an ``UnboundLocalError``, as the definition of ``y`` by ``l()['y'] = 4`` is lost. If the second-to-last line were changed from ``y`` to ``z``, this would be a ``NameError``, as it is today. Keys added to ``frame.f_locals`` that are not lexically local variables remain visible in ``frame.f_locals``, but do not dynamically become local variables. C-API ----- Extensions to the API ''''''''''''''''''''' Three new C-API functions will be added:: PyObject *PyEval_GetFrameLocals(void) PyObject *PyEval_GetFrameGlobals(void) PyObject *PyEval_GetFrameBuiltins(void) ``PyEval_GetFrameLocals()`` is equivalent to: ``locals()``. ``PyEval_GetFrameGlobals()`` is equivalent to: ``globals()``. All these functions will return a new reference. Changes to existing APIs '''''''''''''''''''''''' ``PyFrame_GetLocals(f)`` is equivalent to ``f.f_locals``, and hence its return value will change as described above for accessing ``f.f_locals``. The following C-API functions will be deprecated, as they return borrowed references:: PyEval_GetLocals() PyEval_GetGlobals() PyEval_GetBuiltins() The following functions should be used instead:: PyEval_GetFrameLocals() PyEval_GetFrameGlobals() PyEval_GetFrameBuiltins() which return new references. The semantics of ``PyEval_GetLocals()`` is changed as it now returns a proxy for the frame locals in optimized frames, not a dictionary. The following three functions will become no-ops, and will be deprecated:: PyFrame_FastToLocalsWithError() PyFrame_FastToLocals() PyFrame_LocalsToFast() Behavior of f_locals for optimized functions -------------------------------------------- Although ``f.f_locals`` behaves as if it were the namespace of the function, there will be some observable differences. For example, ``f.f_locals is f.f_locals`` may be ``False``. However ``f.f_locals == f.f_locals`` will be ``True``, and all changes to the underlying variables, by any means, will always be visible. Backwards Compatibility ======================= Python ------ The current implementation has many corner cases and oddities. Code that works around those may need to be changed. Code that uses ``locals()`` for simple templating, or print debugging, will continue to work correctly. Debuggers and other tools that use ``f_locals`` to modify local variables, will now work correctly, even in the presence of threaded code, coroutines and generators. C-API ----- PyEval_GetLocals '''''''''''''''' Because ``PyEval_GetLocals()`` returns a borrowed reference, it requires the proxy mapping to be cached on the frame, extending its lifetime and creating a cycle. ``PyEval_GetFrameLocals()`` should be used instead. This code:: locals = PyEval_GetLocals(); if (locals == NULL) { goto error_handler; } Py_INCREF(locals); should be replaced with:: locals = PyEval_GetFrameLocals(); if (locals == NULL) { goto error_handler; } Implementation ============== Each read of ``frame.f_locals`` will create a new proxy object that gives the appearance of being the mapping of local (including cell and free) variable names to the values of those local variables. A possible implementation is sketched out below. All attributes that start with an underscore are invisible and cannot be accessed directly. They serve only to illustrate the proposed design. :: NULL: Object # NULL is a singleton representing the absence of a value. class CodeType: _name_to_offset_mapping_impl: dict | NULL _cells: frozenset # Set of indexes of cell and free variables ... def __init__(self, ...): self._name_to_offset_mapping_impl = NULL self._variable_names = deduplicate( self.co_varnames + self.co_cellvars + self.co_freevars ) ... @property def _name_to_offset_mapping(self): "Mapping of names to offsets in local variable array." if self._name_to_offset_mapping_impl is NULL: self._name_to_offset_mapping_impl = { name: index for (index, name) in enumerate(self._variable_names) } return self._name_to_offset_mapping_impl class FrameType: _locals : array[Object] # The values of the local variables, items may be NULL. _extra_locals: dict | NULL # Dictionary for storing extra locals not in _locals. _locals_cache: FrameLocalsProxy | NULL # required to support PyEval_GetLocals() def __init__(self, ...): self._extra_locals = NULL self._locals_cache = NULL ... @property def f_locals(self): return FrameLocalsProxy(self) class FrameLocalsProxy: "Implements collections.MutableMapping." __slots__ "_frame" def __init__(self, frame:FrameType): self._frame = frame def __getitem__(self, name): f = self._frame co = f.f_code if name in co._name_to_offset_mapping: index = co._name_to_offset_mapping[name] val = f._locals[index] if val is NULL: raise KeyError(name) if index in co._cells val = val.cell_contents if val is NULL: raise KeyError(name) return val else: if f._extra_locals is NULL: raise KeyError(name) return f._extra_locals[name] def __setitem__(self, name, value): f = self._frame co = f.f_code if name in co._name_to_offset_mapping: index = co._name_to_offset_mapping[name] kind = co._local_kinds[index] if index in co._cells cell = f._locals[index] cell.cell_contents = val else: f._locals[index] = val else: if f._extra_locals is NULL: f._extra_locals = {} f._extra_locals[name] = val def __iter__(self): f = self._frame co = f.f_code yield from iter(f._extra_locals) for index, name in enumerate(co._variable_names): val = f._locals[index] if val is NULL: continue if index in co._cells: val = val.cell_contents if val is NULL: continue yield name def __contains__(self, item): f = self._frame if item in f._extra_locals: return True return item in co._variable_names def __len__(self): f = self._frame co = f.f_code res = 0 for index, _ in enumerate(co._variable_names): val = f._locals[index] if val is NULL: continue if index in co._cells: if val.cell_contents is NULL: continue res += 1 return len(self._extra_locals) + res C API ----- ``PyEval_GetLocals()`` will be implemented roughly as follows:: PyObject *PyEval_GetLocals(void) { PyFrameObject * = ...; // Get the current frame. if (frame->_locals_cache == NULL) { frame->_locals_cache = PyEval_GetFrameLocals(); } return frame->_locals_cache; } As with all functions that return a borrowed reference, care must be taken to ensure that the reference is not used beyond the lifetime of the object. Impact on PEP 709 inlined comprehensions ======================================== For inlined comprehensions within a function, ``locals()`` currently behaves the same inside or outside of the comprehension, and this will not change. The behavior of ``locals()`` inside functions will generally change as specified in the rest of this PEP. For inlined comprehensions at module or class scope, currently calling ``locals()`` within the inlined comprehension returns a new dictionary for each call. This PEP will make ``locals()`` within a function also always return a new dictionary for each call, improving consistency; class or module scope inlined comprehensions will appear to behave as if the inlined comprehension is still a distinct function. Comparison with PEP 558 ======================= This PEP and :pep:`558` share a common goal: to make the semantics of ``locals()`` and ``frame.f_locals()`` intelligible, and their operation reliable. The key difference between this PEP and :pep:`558` is that :pep:`558` keeps an internal copy of the local variables, whereas this PEP does not. :pep:`558` does not specify exactly when the internal copy is updated, making the behavior of :pep:`558` impossible to reason about. Implementation ============== The implementation is in development as a `draft pull request on GitHub `__. Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.