PEP 749: Implementing PEP 649 (#3814)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
This commit is contained in:
Jelle Zijlstra
GitHub
parent
e73affad48
commit
0a803fec89
|
|
@ -624,7 +624,8 @@ peps/pep-0742.rst @JelleZijlstra
|
||||||
peps/pep-0743.rst @vstinner
|
peps/pep-0743.rst @vstinner
|
||||||
peps/pep-0744.rst @brandtbucher
|
peps/pep-0744.rst @brandtbucher
|
||||||
peps/pep-0745.rst @hugovk
|
peps/pep-0745.rst @hugovk
|
||||||
peps/pep-0746.rst @jellezijlstra
|
peps/pep-0746.rst @JelleZijlstra
|
||||||
|
peps/pep-0749.rst @JelleZijlstra
|
||||||
# ...
|
# ...
|
||||||
# peps/pep-0754.rst
|
# peps/pep-0754.rst
|
||||||
# ...
|
# ...
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,753 @@
|
||||||
|
PEP: 749
|
||||||
|
Title: Implementing PEP 649
|
||||||
|
Author: Jelle Zijlstra <jelle.zijlstra@gmail.com>
|
||||||
|
Status: Draft
|
||||||
|
Type: Standards Track
|
||||||
|
Topic: Typing
|
||||||
|
Requires: 649
|
||||||
|
Created: 28-May-2024
|
||||||
|
Python-Version: 3.14
|
||||||
|
|
||||||
|
|
||||||
|
Abstract
|
||||||
|
========
|
||||||
|
|
||||||
|
This PEP supplements :pep:`649` by providing various tweaks and additions to its
|
||||||
|
specification:
|
||||||
|
|
||||||
|
* ``from __future__ import annotations`` (:pep:`563`) will continue to exist with
|
||||||
|
its current behavior at least until Python 3.13 reaches its end-of-life. Subsequently,
|
||||||
|
it will be deprecated and eventually removed.
|
||||||
|
* A new standard library module, ``annotations``, is added to provide tooling for
|
||||||
|
annotations. It will include the ``get_annotations()`` function, an enum for annotation
|
||||||
|
formats, a ``ForwardRef`` class, and a helper function for calling ``__annotate__`` functions.
|
||||||
|
* Annotations in the REPL are lazily evaluated, just like other module-level annotations.
|
||||||
|
* We specify the behavior of wrapper objects that provide annotations, such as :py:func:`classmethod`
|
||||||
|
and code that uses :py:func:`functools.wraps`.
|
||||||
|
* There will not be a code flag for marking ``__annotate__`` functions
|
||||||
|
that can be run in a "fake globals" environment.
|
||||||
|
* Setting the ``__annotations__`` attribute directly will not affect the ``__annotate__`` attribute.
|
||||||
|
* We add functionality to allow evaluating type alias values and type parameter bounds and defaults
|
||||||
|
(which were added by :pep:`695` and :pep:`696`) using PEP 649-like semantics.
|
||||||
|
|
||||||
|
Motivation
|
||||||
|
==========
|
||||||
|
|
||||||
|
:pep:`649` provides an excellent framework for creating better semantics for
|
||||||
|
annotations in Python. It solves a common pain point for users of annotations,
|
||||||
|
including those using static type hints as well as those using runtime typing,
|
||||||
|
and it makes the language more elegant and powerful.
|
||||||
|
The PEP was originally proposed in 2021 for Python 3.10,
|
||||||
|
and it was accepted in 2023. However, the implementation took longer than anticipated,
|
||||||
|
and now the PEP is expected to be implemented in Python 3.14.
|
||||||
|
|
||||||
|
I have started working on the implementation of the PEP in CPython. I found that
|
||||||
|
the PEP leaves some areas underspecified, and some
|
||||||
|
of its decisions in corner cases are questionable. This new PEP proposes several
|
||||||
|
changes and additions to the specification to address these issues.
|
||||||
|
|
||||||
|
This PEP supplements rather than supersedes PEP 649. The changes proposed here
|
||||||
|
should make the overall user experience better, but they do not change the
|
||||||
|
general framework of the earlier PEP.
|
||||||
|
|
||||||
|
|
||||||
|
The future of ``from __future__ import annotations``
|
||||||
|
====================================================
|
||||||
|
|
||||||
|
:pep:`563` previously introduced the future import ``from __future__ import annotations``,
|
||||||
|
which changes all annotations to strings. :pep:`649` proposes an alternative approach
|
||||||
|
that does not require this future import, and states:
|
||||||
|
|
||||||
|
If this PEP is accepted, PEP 563 will be deprecated and eventually removed.
|
||||||
|
|
||||||
|
However, the PEP does not provide a detailed plan for this deprecation.
|
||||||
|
|
||||||
|
There is some previous discussion of this topic `on Discourse <https://discuss.python.org/t/pep-649-deferred-evaluation-of-annotations-tentatively-accepted/21331/44>`__
|
||||||
|
(note that in the linked post I proposed something different from what is proposed here).
|
||||||
|
|
||||||
|
Specification
|
||||||
|
-------------
|
||||||
|
|
||||||
|
We suggest the following deprecation plan:
|
||||||
|
|
||||||
|
- In Python 3.14, ``from __future__ import annotations`` will continue to work as it
|
||||||
|
did before, converting annotations into strings.
|
||||||
|
- Sometime after the last release that did not support :pep:`649` semantics (expected to be 3.13)
|
||||||
|
reaches its end-of-life, ``from __future__ import annotations`` is deprecated. Compiling
|
||||||
|
any code that uses the future import will emit a :py:exc:`DeprecationWarning`. This will
|
||||||
|
happen no sooner than the first release after Python 3.13 reaches its end-of-life, but
|
||||||
|
the community may decide to wait longer.
|
||||||
|
- After at least two releases, the future import is removed, and annotations are always
|
||||||
|
evaluated as per :pep:`649`. Code that continues to use the future import will raise
|
||||||
|
a :py:exc:`SyntaxError`, similar to any other undefined future import.
|
||||||
|
|
||||||
|
Rejected alternatives
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
*Immediately make the future import a no-op*: We considered applying :pep:`649` semantics
|
||||||
|
to all code in Python 3.14, making the future import a no-op. However, this would break
|
||||||
|
code that works in 3.13 under the following set of conditions:
|
||||||
|
|
||||||
|
* ``__future__ import annotations`` is active
|
||||||
|
* There are annotations that rely on forward references
|
||||||
|
* Annotations are eagerly evaluated at import time, for example by a metaclass or
|
||||||
|
class or function decorator. For example, this currently applies to the
|
||||||
|
released version of ``typing_extensions.TypedDict``.
|
||||||
|
|
||||||
|
This is expected to be a common pattern, so we cannot afford to break such code during
|
||||||
|
the upgrade from 3.13 to 3.14.
|
||||||
|
|
||||||
|
Such code would still break when the future import is eventually removed. However, this
|
||||||
|
is many years in the future, giving affected decorators plenty of time to update their code.
|
||||||
|
|
||||||
|
*Immediately deprecate the future import*: Instead of waiting until Python 3.13 reaches
|
||||||
|
its end-of-life, we could immediately start emitting warnings when the future import is
|
||||||
|
used. However, many libraries are already using ``from __future__ import annotations`` as
|
||||||
|
an elegant way to enable unrestricted forward references in their annotations. If we deprecate
|
||||||
|
the future import immediately, it would be impossible for these libraries to use unrestricted
|
||||||
|
forward references on all supported Python versions while avoiding deprecation warnings:
|
||||||
|
unlike other features deprecated from the standard library, a ``__future__`` import must
|
||||||
|
be the first statement in a given module, meaning it would be impossible to only
|
||||||
|
conditionally import ``__future__.annotations`` on Python 3.13 and lower. (The necessary
|
||||||
|
``sys.version_info`` check would count as a statement preceding the ``__future__`` import.)
|
||||||
|
|
||||||
|
*Keep the future import around forever*: We could also decide to keep the future import
|
||||||
|
indefinitely. However, this would permanently bifurcate the behavior of the Python
|
||||||
|
language. This is undesirable; the language should have only a single set of semantics,
|
||||||
|
not two permanently different modes.
|
||||||
|
|
||||||
|
New ``annotations`` module
|
||||||
|
==========================
|
||||||
|
|
||||||
|
:pep:`649` proposes to add tooling related to annotations to the :py:mod:`inspect`
|
||||||
|
module. However, that module is rather large, has direct or indirect dependencies
|
||||||
|
on at least 35 other standard library modules, and is so slow to import that other
|
||||||
|
standard library modules are often discouraged from importing it. Furthermore, we
|
||||||
|
anticipate adding more tools in addition to the :py:func:`inspect.get_annotations`
|
||||||
|
function and the ``VALUE``, ``FORWARDREF``, and ``SOURCE`` formats.
|
||||||
|
|
||||||
|
A new standard library module provides a logical home for this functionality and
|
||||||
|
also enables us to add more tooling that is useful for consumers of annotations.
|
||||||
|
|
||||||
|
Rationale
|
||||||
|
---------
|
||||||
|
|
||||||
|
:pep:`649` indicates that :py:class:`!typing.ForwardRef` should be used to implement the
|
||||||
|
``FORWARDREF`` format in :py:func:`inspect.get_annotations`. However, the existing implementation
|
||||||
|
of :py:class:`!typing.ForwardRef` is intertwined with the rest of the :py:mod:`!typing` module,
|
||||||
|
and it would not make sense to add :py:mod:`!typing`-specific behavior to the generic ``get_annotations()``
|
||||||
|
function. Furthermore, :py:class:`!typing.ForwardRef` is a problematic
|
||||||
|
class: it is public and documented, but the documentation lists no attributes or methods
|
||||||
|
for it. Nonetheless, third-party libraries make use of some of its undocumented
|
||||||
|
attributes. For instance, `Pydantic <https://github.com/pydantic/pydantic/blob/00ff77ed37589d924d3c10e0d5a48a7ef679a0d7/pydantic/v1/typing.py#L66>`__
|
||||||
|
and `Typeguard <https://github.com/agronholm/typeguard/blob/016f8139f5a0a63147d68df9558cc5584cd2c49a/src/typeguard/_utils.py#L44>`__
|
||||||
|
use the ``_evaluate`` method; `beartype <https://github.com/beartype/beartype/blob/0b4453f83c7ed4be054d8733aab8075e1478e166/beartype/_util/hint/pep/proposal/pep484585/utilpep484585ref.py#L210>`__
|
||||||
|
and `pyanalyze <https://github.com/quora/pyanalyze/blob/9e401724f9d035cf138b72612834b6d5a00eb8e8/pyanalyze/annotations.py#L509>`__
|
||||||
|
use the ``__forward_arg__`` attribute.
|
||||||
|
|
||||||
|
We replace the existing but poorly specified :py:class:`!typing.ForwardRef` with a new class,
|
||||||
|
``annotations.ForwardRef``. It is designed to be mostly compatible with existing uses
|
||||||
|
of the :py:class:`!typing.ForwardRef` class, but without the behaviors specific to the
|
||||||
|
:py:mod:`!typing` module. For compatibility with existing users, we keep the private
|
||||||
|
``_evaluate`` method, but mark it as deprecated. It delegates to a new public function in
|
||||||
|
the :py:mod:`!typing` module, ``typing.evaluate_forward_ref``, that is designed to
|
||||||
|
evaluate forward references in a way that is specific to type hints.
|
||||||
|
|
||||||
|
We add a function ``annotations.call_annotate_function`` as a helper for calling
|
||||||
|
``__annotate__`` functions. This is a useful building block when implementing functionality
|
||||||
|
that needs to partially evaluate annotations while a class is being constructed.
|
||||||
|
For example, the implementation of :py:class:`typing.NamedTuple` needs to retrieve
|
||||||
|
the annotations from a class namespace dictionary before the namedtuple class itself
|
||||||
|
can be constructed, because the annotations determine what fields exist on the namedtuple.
|
||||||
|
|
||||||
|
Specification
|
||||||
|
-------------
|
||||||
|
|
||||||
|
A new module, ``annotations``, is added to the standard library. Its aim is to
|
||||||
|
provide tooling for introspecting and wrapping annotations.
|
||||||
|
|
||||||
|
The exact contents of the module are not yet specified. We will add support for
|
||||||
|
:pep:`649` semantics to standard library functionality that uses annotations, such
|
||||||
|
as :py:mod:`dataclasses` and :py:class:`typing.TypedDict`, and use the experience
|
||||||
|
to inform the design of the new module.
|
||||||
|
|
||||||
|
The module will contain the following functionality:
|
||||||
|
|
||||||
|
* ``get_annotations()``: A function that returns the annotations of a function,
|
||||||
|
module, or class. This will replace :py:func:`inspect.get_annotations`. The latter
|
||||||
|
will delegate to the new function. It may eventually be deprecated, but to
|
||||||
|
minimize disruption, we do not propose an immediate deprecation.
|
||||||
|
* ``Format``: an enum that contains the possible formats of annotations. This will
|
||||||
|
replace the ``VALUE``, ``FORWARDREF``, and ``SOURCE`` formats in :pep:`649`.
|
||||||
|
PEP 649 proposed to make these values global members of the :py:mod:`inspect`
|
||||||
|
module; we prefer to place them within an enum.
|
||||||
|
* ``ForwardRef``: a class representing a forward reference; it may be returned by
|
||||||
|
``get_annotations()`` when the format is ``FORWARDREF``. The existing class
|
||||||
|
:py:class:`typing.ForwardRef` will become an alias of this class. Its members include:
|
||||||
|
|
||||||
|
* ``__forward_arg__``: the string argument of the forward reference
|
||||||
|
* ``evaluate(globals=None, locals=None, type_params=None, owner=None)``: a method that attempts to evaluate
|
||||||
|
the forward reference. The ``ForwardRef`` object may hold a reference to the
|
||||||
|
globals and other namespaces of the object that it originated from. If so, these
|
||||||
|
namespaces may be used to evaluate the forward reference. The *owner* argument
|
||||||
|
may be the object that holds the original annotation, such as the class or module
|
||||||
|
object; it is used to extract the globals and locals namespaces if these are not
|
||||||
|
provided.
|
||||||
|
* ``_evaluate()``, with the same interface as the existing ``ForwardRef._evaluate``
|
||||||
|
method. It will be undocumented and immediately deprecated. It is provided for
|
||||||
|
compatibility with existing users of ``typing.ForwardRef``.
|
||||||
|
|
||||||
|
* ``call_annotate_function(func: Callable, format: Format)``: a helper for calling
|
||||||
|
an ``__annotate__`` function with a given format. If the function does not support
|
||||||
|
this format, ``call_annotate_function()`` will set up a "fake globals" environment,
|
||||||
|
as described in :pep:`649`, and use that environment to return the desired annotations
|
||||||
|
format.
|
||||||
|
* ``call_evaluate_function(func: Callable | None, format: Format)``: similar to
|
||||||
|
``call_annotate_function``, but does not rely on the function returning an annotations
|
||||||
|
dictionary. This is intended to be used for evaluating deferred attributes introduced by
|
||||||
|
:pep:`695` and :pep:`696`; see below for details. *func* may be ``None``
|
||||||
|
for convenience; if ``None`` is passed, the function also returns ``None``.
|
||||||
|
|
||||||
|
A new function is also added to the :py:mod:`!typing` module, ``typing.evaluate_forward_ref``.
|
||||||
|
This function is a wrapper around the ``ForwardRef.evaluate`` method, but it performs
|
||||||
|
additional work that is specific to type hints. For example, it recurses into complex
|
||||||
|
types and evaluates additional forward references within these types.
|
||||||
|
|
||||||
|
Open issues
|
||||||
|
-----------
|
||||||
|
|
||||||
|
What should this module be called? Some ideas:
|
||||||
|
|
||||||
|
- ``annotations``: The most obvious name, but it may cause confusion with the existing
|
||||||
|
``from __future__ import annotations``. There is a PyPI package :pypi:`annotations`,
|
||||||
|
but it had only a single release in 2015 and looks abandoned.
|
||||||
|
- ``annotools``: Analogous to :py:mod:`itertools` and :py:mod:`functools`, but "anno" is a less
|
||||||
|
obvious abbreviation than "iter" or "func". As of this writing, there
|
||||||
|
is no PyPI package with this name.
|
||||||
|
- ``annotationtools``: A more explicit version. There is a PyPI package
|
||||||
|
:pypi:`annotationtools`, which had a release in 2023.
|
||||||
|
- ``annotation_tools``: A variation of the above but without a PyPI conflict. However,
|
||||||
|
no other public standard library module has an underscore in its name.
|
||||||
|
- ``annotationslib``: Analogous to :py:mod:`tomllib`, :py:mod:`pathlib`, and :py:mod:`importlib`.
|
||||||
|
There is no PyPI package with this name.
|
||||||
|
|
||||||
|
Rejected alternatives
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
*Add the functionality to the inspect module*: As described above, the
|
||||||
|
:py:mod:`inspect` module is already quite large, and its import time is prohibitive
|
||||||
|
for some use cases.
|
||||||
|
|
||||||
|
*Add the functionality to the typing module*: While annotations are mostly
|
||||||
|
used for typing, they may also be used for other purposes. We prefer to keep a clean
|
||||||
|
separation between functionality for introspecting annotations and functionality that
|
||||||
|
is exclusively meant for type hints.
|
||||||
|
|
||||||
|
*Add the functionality to the types module*: The :py:mod:`types` module is
|
||||||
|
meant for functionality related to *types*, and annotations can exist on functions
|
||||||
|
and modules, not only on types.
|
||||||
|
|
||||||
|
*Develop this functionality in a third-party package*: The functionality in this new
|
||||||
|
module will be pure Python code, and it is possible to implement a third-party package
|
||||||
|
that provides the same functionality by interacting directly with ``__annotate__``
|
||||||
|
functions generated by the interpreter. However, the functionality of the proposed new
|
||||||
|
module will certainly be useful in the standard library itself (e.g., for implementing
|
||||||
|
:py:mod:`dataclasses` and :py:class:`typing.NamedTuple`), so it makes sense to include
|
||||||
|
it in the standard library.
|
||||||
|
|
||||||
|
*Add this functionality to a private module*: It would be possible to initially develop
|
||||||
|
the module in a private standard library module (e.g., ``_annotations``), and publicize
|
||||||
|
it only after we have gained more experience with the API. However, we already know
|
||||||
|
that we will need parts of this module for the standard library itself (e.g., for
|
||||||
|
implementing :py:mod:`!dataclasses` and :py:class:`!typing.NamedTuple`). Even if we make
|
||||||
|
it private, the module will inevitably get used by third-party users. It is preferable
|
||||||
|
to start with a clear, documented API from the beginning, to enable third-party users to
|
||||||
|
support :pep:`649` semantics as thoroughly as the standard library. The module will
|
||||||
|
immediately be used in other parts of the standard library, ensuring that it covers a
|
||||||
|
reasonable set of use cases.
|
||||||
|
|
||||||
|
Behavior of the REPL
|
||||||
|
====================
|
||||||
|
|
||||||
|
:pep:`649` specifies the following behavior of the interactive REPL:
|
||||||
|
|
||||||
|
For the sake of simplicity, in this case we forego delayed evaluation.
|
||||||
|
Module-level annotations in the REPL shell will continue to work exactly
|
||||||
|
as they do with “stock semantics”, evaluating immediately and setting the
|
||||||
|
result directly inside the ``__annotations__`` dict.
|
||||||
|
|
||||||
|
There are several problems with this proposed behavior. It makes the REPL the
|
||||||
|
only context where annotations are still evaluated immediately, which is
|
||||||
|
confusing for users and complicates the language.
|
||||||
|
|
||||||
|
It also makes the implementation of the REPL more complex, as it needs to
|
||||||
|
ensure that all statements are compiled in "interactive" mode, even if their
|
||||||
|
output does not need to be displayed. (This matters if there are multiple
|
||||||
|
statements in a single line evaluated by the REPL.)
|
||||||
|
|
||||||
|
Most importantly, this breaks some plausible use cases that inexperienced
|
||||||
|
users could run into. A user might write the following in a file::
|
||||||
|
|
||||||
|
a: X | None = None
|
||||||
|
class X: ...
|
||||||
|
|
||||||
|
Under :pep:`649` this would work fine: ``X`` is not yet defined when it is used
|
||||||
|
in the annotation for ``a``, but the annotation is lazily evaluated. However,
|
||||||
|
if a user were to paste this same code into the REPL and execute it line by
|
||||||
|
line, it would throw a ``NameError``, because the name ``X`` is not yet defined.
|
||||||
|
|
||||||
|
This topic was previously discussed `on Discourse <https://discuss.python.org/t/pep-649-behavior-of-the-repl/54109>`__.
|
||||||
|
|
||||||
|
Specification
|
||||||
|
-------------
|
||||||
|
|
||||||
|
We propose to treat the interactive console like any other module-level code, and
|
||||||
|
make annotations lazily evaluated. This makes the language more consistent and
|
||||||
|
avoids subtle behavior changes between modules and the REPL.
|
||||||
|
|
||||||
|
Because the REPL is evaluated line by line, we would generate a new ``__annotate__``
|
||||||
|
function for every evaluated statement in the global scope that contains annotations. Whenever a line
|
||||||
|
containing annotations is evaluated, the previous ``__annotate__`` function is
|
||||||
|
lost:
|
||||||
|
|
||||||
|
.. code:: pycon
|
||||||
|
|
||||||
|
>>> x: int
|
||||||
|
>>> __annotate__(1)
|
||||||
|
{'x': <class 'int'>}
|
||||||
|
>>> y: str
|
||||||
|
>>> __annotate__(1)
|
||||||
|
{'y': <class 'str'>}
|
||||||
|
>>> z: doesntexist
|
||||||
|
>>> __annotate__(1)
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "<python-input-5>", line 1, in <module>
|
||||||
|
__annotate__(1)
|
||||||
|
~~~~~~~~~~~~^^^
|
||||||
|
File "<python-input-4>", line 1, in __annotate__
|
||||||
|
z: doesntexist
|
||||||
|
^^^^^^^^^^^
|
||||||
|
NameError: name 'doesntexist' is not defined
|
||||||
|
|
||||||
|
There will be no ``__annotations__`` key in the global namespace of the REPL.
|
||||||
|
In module namespaces, this key is created lazily when the ``__annotations__``
|
||||||
|
descriptor of the module object is accessed, but in the REPL there is no such module
|
||||||
|
object.
|
||||||
|
|
||||||
|
Classes and functions defined within the REPL will also work like any other classes,
|
||||||
|
so evaluation of their annotations will be deferred. It is possible to access the
|
||||||
|
``__annotations__`` and ``__annotate__`` attributes or use the ``annotations`` module
|
||||||
|
to introspect the annotations.
|
||||||
|
|
||||||
|
Wrappers that provide ``__annotations__``
|
||||||
|
=========================================
|
||||||
|
|
||||||
|
Several objects in the standard library and elsewhere provide annotations for their
|
||||||
|
wrapped object. :pep:`649` does not specify how such wrappers should behave.
|
||||||
|
|
||||||
|
Specification
|
||||||
|
-------------
|
||||||
|
|
||||||
|
Wrappers that provide annotations should be designed with the following goals
|
||||||
|
in mind:
|
||||||
|
|
||||||
|
* Evaluation of ``__annotations__`` should be deferred for as long as possible,
|
||||||
|
consistent with the behavior of built-in functions, classes, and modules.
|
||||||
|
* Backward compatibility with the behavior prior to the implementation of :pep:`649`
|
||||||
|
should be preserved.
|
||||||
|
* The ``__annotate__`` and ``__annotations__`` attributes should both be supplied
|
||||||
|
with semantics consistent to those of the wrapped object.
|
||||||
|
|
||||||
|
More specifically:
|
||||||
|
|
||||||
|
* :py:func:`functools.update_wrapper` (and therefore :py:func:`functools.wraps`)
|
||||||
|
will copy only the ``__annotate__`` attribute
|
||||||
|
from the wrapped object to the wrapper. The ``__annotations__`` descriptor on the
|
||||||
|
wrapper function will use the copied ``__annotate__``.
|
||||||
|
* The constructors for :py:func:`classmethod` and :py:func:`staticmethod` currently
|
||||||
|
copy the ``__annotations__`` attribute from the wrapped object to the wrapper.
|
||||||
|
They will instead have writable attributes for
|
||||||
|
``__annotate__`` and ``__annotations__``. Reading these attributes will retrieve
|
||||||
|
the corresponding attribute from the underlying callable and cache it in the wrapper's
|
||||||
|
``__dict__``. Writing to these attributes will directly update the ``__dict__``,
|
||||||
|
without affecting the wrapped callable.
|
||||||
|
|
||||||
|
Remove code flag for marking ``__annotate__`` functions
|
||||||
|
=======================================================
|
||||||
|
|
||||||
|
:pep:`649` specifies:
|
||||||
|
|
||||||
|
This PEP assumes that
|
||||||
|
third-party libraries may implement their own ``__annotate__``
|
||||||
|
methods, and those functions would almost certainly work
|
||||||
|
incorrectly when run in this "fake globals" environment.
|
||||||
|
For that reason, this PEP allocates a flag on code objects,
|
||||||
|
one of the unused bits in ``co_flags``, to mean "This code
|
||||||
|
object can be run in a 'fake globals' environment." This
|
||||||
|
makes the "fake globals" environment strictly opt-in, and
|
||||||
|
it's expected that only ``__annotate__`` methods generated
|
||||||
|
by the Python compiler will set it.
|
||||||
|
|
||||||
|
We have not found a need for this mechanism during our work to
|
||||||
|
add :pep:`649` support to the standard library. While it is true
|
||||||
|
that custom ``__annotate__`` functions may not work well with the
|
||||||
|
"fake globals" environment, this technique is used only when the
|
||||||
|
``__annotate__`` function raises :py:exc:`NotImplementedError` to
|
||||||
|
signal that it does not support the requested format. However,
|
||||||
|
manually implemented ``__annotate__`` functions are likely to support
|
||||||
|
all three annotation formats; often, they will consist of a call to
|
||||||
|
``annotations.call_annotate_function`` plus some transformation of the
|
||||||
|
result.
|
||||||
|
|
||||||
|
In addition, the proposed mechanism couples the implementation with
|
||||||
|
low-level details of the code object. The code object flags are
|
||||||
|
CPython-specific and the documentation :py:ref:`explicitly warns <inspect-module-co-flags>`
|
||||||
|
against relying on the values.
|
||||||
|
|
||||||
|
Specification
|
||||||
|
-------------
|
||||||
|
|
||||||
|
The standard library will use the "fake globals" technique on any
|
||||||
|
``__annotate__`` function that raises :py:exc:`NotImplementedError`
|
||||||
|
when the requested format is not supported.
|
||||||
|
|
||||||
|
Third-party code that implements ``__annotate__`` functions should either
|
||||||
|
support all three annotation formats, or be prepared to handle the
|
||||||
|
"fake globals" environment. This should be mentioned in the data model
|
||||||
|
documentation for ``__annotate__``.
|
||||||
|
|
||||||
|
Effect of setting ``__annotations__``
|
||||||
|
=====================================
|
||||||
|
|
||||||
|
:pep:`649` specifies:
|
||||||
|
|
||||||
|
Setting ``o.__annotations__`` to a legal value
|
||||||
|
automatically sets ``o.__annotate__`` to ``None``.
|
||||||
|
|
||||||
|
We would prefer to keep ``__annotate__`` unchanged when ``__annotations__``
|
||||||
|
is written to. Conceptually, ``__annotate__`` provides the ground truth
|
||||||
|
and ``__annotations__`` is merely a cache, and we shouldn't throw away the
|
||||||
|
ground truth if the cache is modified.
|
||||||
|
|
||||||
|
The motivation for :pep:`649`'s behavior is to keep the two attributes in sync.
|
||||||
|
However, this is impossible in general; if the ``__annotations__`` dictionary
|
||||||
|
is modified in place, this will not be reflected in the ``__annotate__`` attribute.
|
||||||
|
The overall mental model for this area will be simpler if setting ``__annotations__``
|
||||||
|
has no effect on ``__annotate__``.
|
||||||
|
|
||||||
|
Specification
|
||||||
|
-------------
|
||||||
|
|
||||||
|
The value of ``__annotate__`` is not changed when ``__annotations__`` is set.
|
||||||
|
|
||||||
|
Deferred evaluation of PEP 695 and 696 objects
|
||||||
|
==============================================
|
||||||
|
|
||||||
|
Since :pep:`649` was written, Python 3.12 and 3.13 gained support for
|
||||||
|
several new features that also use deferred evaluation, similar to the
|
||||||
|
behavior this PEP proposes for annotations:
|
||||||
|
|
||||||
|
* The value of type aliases created through the :py:keyword:`type`
|
||||||
|
statement (:pep:`695`)
|
||||||
|
* The bound and constraints of :py:class:`typing.TypeVar` objects
|
||||||
|
created through the syntax for generics (:pep:`695`)
|
||||||
|
* The default value of :py:class:`typing.TypeVar`, :py:class:`ParamSpec`,
|
||||||
|
and :py:class:`typing.TypeVarTuple` objects (:pep:`696`)
|
||||||
|
|
||||||
|
Currently, these objects use deferred evaluation, but there is no direct
|
||||||
|
access to the function object used for deferred evaluation. To enable
|
||||||
|
the same kind of introspection that is now possible for annotations, we propose
|
||||||
|
to expose the internal function objects, allowing users to evaluate them
|
||||||
|
using the FORWARDREF and SOURCE formats.
|
||||||
|
|
||||||
|
Specification
|
||||||
|
-------------
|
||||||
|
|
||||||
|
We will add the following new attributes:
|
||||||
|
|
||||||
|
* ``evaluate_value`` on :py:class:`typing.TypeAliasType`
|
||||||
|
* ``evaluate_bound``, ``evaluate_constraints``, and ``evaluate_default`` on :py:class:`typing.TypeVar`
|
||||||
|
* ``evaluate_default`` on :py:class:`typing.ParamSpec`
|
||||||
|
* ``evaluate_default`` on :py:class:`typing.TypeVarTuple`
|
||||||
|
|
||||||
|
Except for ``evaluate_value``, these attributes may be ``None`` if the object
|
||||||
|
does not have a bound, constraints, or default. Otherwise, the attribute is a
|
||||||
|
callable, similar to an ``__annotate__`` function, that takes a single integer
|
||||||
|
argument and returns the evaluated value. Unlike ``__annotate__`` functions,
|
||||||
|
these callables return a single value, not a dictionary of annotations.
|
||||||
|
These attributes are read-only.
|
||||||
|
|
||||||
|
Usually, users would use these attributes in combinations with
|
||||||
|
``annotations.call_evaluate_function``. For example, to get a ``TypeVar``'s bound
|
||||||
|
in SOURCE format, one could write
|
||||||
|
``annotations.call_evaluate_function(T.evaluate_bound, annotations.Format.SOURCE)``.
|
||||||
|
|
||||||
|
Miscellaneous implementation details
|
||||||
|
====================================
|
||||||
|
|
||||||
|
:pep:`649` goes into considerable detail on some aspects of the implementation.
|
||||||
|
To avoid confusion, we describe a few aspects where the current implementation
|
||||||
|
differs from that described in the PEP. However, these details are not guaranteed
|
||||||
|
to hold in the future, and they may change without notice in the future, unless
|
||||||
|
they are documented in the language reference.
|
||||||
|
|
||||||
|
Supported operations on ``ForwardRef`` objects
|
||||||
|
----------------------------------------------
|
||||||
|
|
||||||
|
The ``SOURCE`` format is implemented by the "stringizer" technique,
|
||||||
|
where the globals dictionary of a function is augmented so that every
|
||||||
|
lookup results in a special object that can be used to reconstruct the
|
||||||
|
operations that are performed on the object.
|
||||||
|
|
||||||
|
:pep:`649` specifies:
|
||||||
|
|
||||||
|
In practice, the "stringizer" functionality will be implemented
|
||||||
|
in the ``ForwardRef`` object currently defined in the
|
||||||
|
``typing`` module. ``ForwardRef`` will be extended to
|
||||||
|
implement all stringizer functionality; it will also be
|
||||||
|
extended to support evaluating the string it contains,
|
||||||
|
to produce the real value (assuming all symbols referenced
|
||||||
|
are defined).
|
||||||
|
|
||||||
|
However, this is likely to lead to confusion in practice. An object
|
||||||
|
that implements stringizer functionality must implement almost all
|
||||||
|
special methods, including ``__getattr__`` and ``__eq__``, to return
|
||||||
|
a new stringizer. Such an object is confusing to work with: all operations
|
||||||
|
succeed, but they are likely to return different objects than the user
|
||||||
|
expects.
|
||||||
|
|
||||||
|
The current implementation instead implements only a few useful methods
|
||||||
|
on the ``ForwardRef`` class. During the evaluation of annotations,
|
||||||
|
an instance of a private stringizer class is used instead of ``ForwardRef``.
|
||||||
|
After evaluation completes, the implementation of the FORWARDREF format
|
||||||
|
converts these internal objects into ``ForwardRef`` objects.
|
||||||
|
|
||||||
|
Backwards Compatibility
|
||||||
|
=======================
|
||||||
|
|
||||||
|
:pep:`649` provides a thorough discussion of the backwards compatibility implications
|
||||||
|
on existing code that uses either stock or :pep:`563` semantics.
|
||||||
|
|
||||||
|
However, there is another set of compatibility problems: new code that is written
|
||||||
|
assuming :pep:`649` semantics, but uses existing tools that eagerly evaluate annotations.
|
||||||
|
For example, consider a ``dataclass``-like class decorator ``@annotator`` that retrieves the annotated
|
||||||
|
fields in the class it decorates, either by accessing ``__annotations__`` directly
|
||||||
|
or by calling :py:func:`inspect.get_annotations`.
|
||||||
|
|
||||||
|
Once :pep:`649` is implemented, code like this will work fine::
|
||||||
|
|
||||||
|
class X:
|
||||||
|
y: Y
|
||||||
|
|
||||||
|
class Y: pass
|
||||||
|
|
||||||
|
But this will not, unless ``@annotator`` is changed to use the new ``FORWARDREF``
|
||||||
|
format::
|
||||||
|
|
||||||
|
@annotator
|
||||||
|
class X:
|
||||||
|
y: Y
|
||||||
|
|
||||||
|
class Y: pass
|
||||||
|
|
||||||
|
This is not strictly a backwards compatibility issue, since no previously working code
|
||||||
|
would break; before :pep:`649`, this code would have raised ``NameError`` at runtime.
|
||||||
|
In a sense, it is no different from any other new Python feature that needs
|
||||||
|
to be supported by third-party libraries. Nevertheless, it is a serious issue for libraries
|
||||||
|
that perform introspection, and it is important that we make it as easy as possible for
|
||||||
|
libraries to support the new semantics in a straightforward, user-friendly way.
|
||||||
|
|
||||||
|
We will update those parts of the standard library that are affected by this problem,
|
||||||
|
and we propose to add commonly useful functionality to the new ``annotations`` module,
|
||||||
|
so third-party tools can use the same set of tools.
|
||||||
|
|
||||||
|
|
||||||
|
Security Implications
|
||||||
|
=====================
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
|
||||||
|
How to Teach This
|
||||||
|
=================
|
||||||
|
|
||||||
|
The semantics of :pep:`649`, as modified by this PEP, should largely be intuitive for
|
||||||
|
users who add annotations to their code. We eliminate the need for manually adding
|
||||||
|
quotes around annotations that require forward references, a major source of confusion
|
||||||
|
for users.
|
||||||
|
|
||||||
|
For advanced users who need to introspect annotations, the story becomes more complex.
|
||||||
|
The documentation of the new ``annotations`` module will serve as a reference for users
|
||||||
|
who need to interact programmatically with annotations.
|
||||||
|
|
||||||
|
|
||||||
|
Reference Implementation
|
||||||
|
========================
|
||||||
|
|
||||||
|
The in-progress PR `#119891 <https://github.com/python/cpython/pull/119891>`__
|
||||||
|
implements much of this PEP.
|
||||||
|
|
||||||
|
Open Issues
|
||||||
|
===========
|
||||||
|
|
||||||
|
We may discover additional areas where :pep:`649` needs clarification or amendment
|
||||||
|
as we make progress on implementing it. Readers are encouraged to follow the
|
||||||
|
`CPython issue <https://github.com/python/cpython/issues/119180>`__ tracking the
|
||||||
|
implementation of the PEP and try out the draft implementation. Any feedback may
|
||||||
|
be incorporated into future versions of this PEP.
|
||||||
|
|
||||||
|
Should dataclass field types use deferred evaluation?
|
||||||
|
-----------------------------------------------------
|
||||||
|
|
||||||
|
The current draft implementation already supports deferred evaluation in dataclasses,
|
||||||
|
so this works:
|
||||||
|
|
||||||
|
.. code:: pycon
|
||||||
|
|
||||||
|
>>> from dataclasses import dataclass
|
||||||
|
>>> @dataclass
|
||||||
|
... class D:
|
||||||
|
... x: undefined
|
||||||
|
...
|
||||||
|
|
||||||
|
However, the ``FORWARDREF`` format leaks into the field types of the dataclass:
|
||||||
|
|
||||||
|
.. code:: pycon
|
||||||
|
|
||||||
|
>>> fields(D)[0].type
|
||||||
|
ForwardRef('undefined')
|
||||||
|
|
||||||
|
We could instead add deferred evaluation for the field type, similar to that outlined
|
||||||
|
above for type alias values.
|
||||||
|
|
||||||
|
Accessing ``.type`` might throw an error:
|
||||||
|
|
||||||
|
.. code:: pycon
|
||||||
|
|
||||||
|
>>> @dataclass
|
||||||
|
... class D:
|
||||||
|
... x: undefined
|
||||||
|
...
|
||||||
|
>>> field = fields(D)[0]
|
||||||
|
>>> field.type
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "<python-input-4>", line 1, in <module>
|
||||||
|
field.type
|
||||||
|
File ".../dataclasses.py", line 308, in type
|
||||||
|
annos = self._annotate(annotations.Format.VALUE)
|
||||||
|
File "<python-input-2>", line 3, in __annotate__
|
||||||
|
x: undefined
|
||||||
|
^^^^^^^^^
|
||||||
|
NameError: name 'undefined' is not defined
|
||||||
|
|
||||||
|
But users could use ``annotations.call_evaluate_function`` to get the type in other formats:
|
||||||
|
|
||||||
|
.. code:: pycon
|
||||||
|
|
||||||
|
>>> annotations.call_evaluate_function(field.evaluate_type, annotations.Format.SOURCE)
|
||||||
|
'undefined'
|
||||||
|
>>> annotations.call_evaluate_function(field.evaluate_type, annotations.Format.FORWARDREF)
|
||||||
|
ForwardRef('undefined')
|
||||||
|
|
||||||
|
Other variations are possible. For example, we could leave the ``type`` attribute unchanged,
|
||||||
|
and only add the ``evaluate_type`` method. This avoids unpleasant surprises where accessing
|
||||||
|
``.type`` may throw an exception.
|
||||||
|
|
||||||
|
Acknowledgments
|
||||||
|
===============
|
||||||
|
|
||||||
|
First of all, I thank Larry Hastings for writing :pep:`649`. This PEP modifies some of his
|
||||||
|
initial decisions, but the overall design is still his.
|
||||||
|
|
||||||
|
I thank Carl Meyer and Alex Waygood for feedback on early drafts of this PEP.
|
||||||
|
|
||||||
|
Appendix
|
||||||
|
========
|
||||||
|
|
||||||
|
Which expressions can be stringified?
|
||||||
|
-------------------------------------
|
||||||
|
|
||||||
|
:pep:`649` acknowledges that the stringifier cannot handle all expressions. Now that we
|
||||||
|
have a draft implementation, we can be more precise about the expressions that can and
|
||||||
|
cannot be handled. Below is a list of all expressions in the Python AST that can and
|
||||||
|
cannot be recovered by the stringifier. The full list should probably not be added to
|
||||||
|
the documentation, but creating it is a useful exercise.
|
||||||
|
|
||||||
|
First, the stringifier of course cannot recover any information that is not present in
|
||||||
|
the compiled code, including comments, whitespace, parenthesization, and operations that
|
||||||
|
get simplified by the AST optimizer.
|
||||||
|
|
||||||
|
Second, the stringifier can intercept almost all operations that involve names looked
|
||||||
|
up in some scope, but it cannot intercept operations that operate fully on constants.
|
||||||
|
As a corollary, this also means it is not safe to request the ``SOURCE`` format on
|
||||||
|
untrusted code: Python is powerful enough that it is possible to achieve arbitrary
|
||||||
|
code execution even with no access to any globals or builtins. For example:
|
||||||
|
|
||||||
|
.. code-block:: pycon
|
||||||
|
|
||||||
|
>>> def f(x: (1).__class__.__base__.__subclasses__()[-1].__init__.__builtins__["print"]("Hello world")): pass
|
||||||
|
...
|
||||||
|
>>> annotations.get_annotations(f, format=annotations.Format.SOURCE)
|
||||||
|
Hello world
|
||||||
|
{'x': 'None'}
|
||||||
|
|
||||||
|
(This particular example worked for me on the current implementation of a draft of this PEP;
|
||||||
|
the exact code may not keep working in the future.)
|
||||||
|
|
||||||
|
The following are supported (sometimes with caveats):
|
||||||
|
|
||||||
|
* ``BinOp``
|
||||||
|
* ``UnaryOp``
|
||||||
|
|
||||||
|
* ``Invert`` (``~``), ``UAdd`` (``+``), and ``USub`` (``-``) are supported
|
||||||
|
* ``Not`` (``not``) is not supported
|
||||||
|
|
||||||
|
* ``Dict`` (except when using ``**`` unpacking)
|
||||||
|
* ``Set``
|
||||||
|
* ``Compare``
|
||||||
|
|
||||||
|
* ``Eq`` and ``NotEq`` are supported
|
||||||
|
* ``Lt``, ``LtE``, ``Gt``, and ``GtE`` are supported, but the operand may be flipped
|
||||||
|
* ``Is``, ``IsNot``, ``In``, and ``NotIn`` are not supported
|
||||||
|
|
||||||
|
* ``Call`` (except when using ``**`` unpacking)
|
||||||
|
* ``Constant`` (though not the exact representation of the constant; for example, escape
|
||||||
|
sequences in strings are lost; hexadecimal numbers are converted to decimal)
|
||||||
|
* ``Attribute`` (assuming the value is not a constant)
|
||||||
|
* ``Subscript`` (assuming the value is not a constant)
|
||||||
|
* ``Starred`` (``*`` unpacking)
|
||||||
|
* ``Name``
|
||||||
|
* ``List``
|
||||||
|
* ``Tuple``
|
||||||
|
* ``Slice``
|
||||||
|
|
||||||
|
The following are unsupported, but throw an informative error when encountered by the
|
||||||
|
stringifier:
|
||||||
|
|
||||||
|
* ``FormattedValue`` (f-strings; error is not detected if conversion specifiers like ``!r``
|
||||||
|
are used)
|
||||||
|
* ``JoinedStr`` (f-strings)
|
||||||
|
|
||||||
|
The following are unsupported and result in incorrect output:
|
||||||
|
|
||||||
|
* ``BoolOp`` (``and`` and ``or``)
|
||||||
|
* ``IfExp``
|
||||||
|
* ``Lambda``
|
||||||
|
* ``ListComp``
|
||||||
|
* ``SetComp``
|
||||||
|
* ``DictComp``
|
||||||
|
* ``GeneratorExp``
|
||||||
|
|
||||||
|
The following are disallowed in annotation scopes and therefore not relevant:
|
||||||
|
|
||||||
|
* ``NamedExpr`` (``:=``)
|
||||||
|
* ``Await``
|
||||||
|
* ``Yield``
|
||||||
|
* ``YieldFrom``
|
||||||
|
|
||||||
|
|
||||||
|
Copyright
|
||||||
|
=========
|
||||||
|
|
||||||
|
This document is placed in the public domain or under the
|
||||||
|
CC0-1.0-Universal license, whichever is more permissive.
|
||||||
Loading…
Reference in New Issue