PEP 749: Implementing PEP 649 (#3814)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
This commit is contained in:
Jelle Zijlstra
GitHub
parent
e73affad48
commit
0a803fec89
|
|
@ -624,7 +624,8 @@ peps/pep-0742.rst @JelleZijlstra
|
|||
peps/pep-0743.rst @vstinner
|
||||
peps/pep-0744.rst @brandtbucher
|
||||
peps/pep-0745.rst @hugovk
|
||||
peps/pep-0746.rst @jellezijlstra
|
||||
peps/pep-0746.rst @JelleZijlstra
|
||||
peps/pep-0749.rst @JelleZijlstra
|
||||
# ...
|
||||
# peps/pep-0754.rst
|
||||
# ...
|
||||
|
|
|
|||
|
|
@ -0,0 +1,753 @@
|
|||
PEP: 749
|
||||
Title: Implementing PEP 649
|
||||
Author: Jelle Zijlstra <jelle.zijlstra@gmail.com>
|
||||
Status: Draft
|
||||
Type: Standards Track
|
||||
Topic: Typing
|
||||
Requires: 649
|
||||
Created: 28-May-2024
|
||||
Python-Version: 3.14
|
||||
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
This PEP supplements :pep:`649` by providing various tweaks and additions to its
|
||||
specification:
|
||||
|
||||
* ``from __future__ import annotations`` (:pep:`563`) will continue to exist with
|
||||
its current behavior at least until Python 3.13 reaches its end-of-life. Subsequently,
|
||||
it will be deprecated and eventually removed.
|
||||
* A new standard library module, ``annotations``, is added to provide tooling for
|
||||
annotations. It will include the ``get_annotations()`` function, an enum for annotation
|
||||
formats, a ``ForwardRef`` class, and a helper function for calling ``__annotate__`` functions.
|
||||
* Annotations in the REPL are lazily evaluated, just like other module-level annotations.
|
||||
* We specify the behavior of wrapper objects that provide annotations, such as :py:func:`classmethod`
|
||||
and code that uses :py:func:`functools.wraps`.
|
||||
* There will not be a code flag for marking ``__annotate__`` functions
|
||||
that can be run in a "fake globals" environment.
|
||||
* Setting the ``__annotations__`` attribute directly will not affect the ``__annotate__`` attribute.
|
||||
* We add functionality to allow evaluating type alias values and type parameter bounds and defaults
|
||||
(which were added by :pep:`695` and :pep:`696`) using PEP 649-like semantics.
|
||||
|
||||
Motivation
|
||||
==========
|
||||
|
||||
:pep:`649` provides an excellent framework for creating better semantics for
|
||||
annotations in Python. It solves a common pain point for users of annotations,
|
||||
including those using static type hints as well as those using runtime typing,
|
||||
and it makes the language more elegant and powerful.
|
||||
The PEP was originally proposed in 2021 for Python 3.10,
|
||||
and it was accepted in 2023. However, the implementation took longer than anticipated,
|
||||
and now the PEP is expected to be implemented in Python 3.14.
|
||||
|
||||
I have started working on the implementation of the PEP in CPython. I found that
|
||||
the PEP leaves some areas underspecified, and some
|
||||
of its decisions in corner cases are questionable. This new PEP proposes several
|
||||
changes and additions to the specification to address these issues.
|
||||
|
||||
This PEP supplements rather than supersedes PEP 649. The changes proposed here
|
||||
should make the overall user experience better, but they do not change the
|
||||
general framework of the earlier PEP.
|
||||
|
||||
|
||||
The future of ``from __future__ import annotations``
|
||||
====================================================
|
||||
|
||||
:pep:`563` previously introduced the future import ``from __future__ import annotations``,
|
||||
which changes all annotations to strings. :pep:`649` proposes an alternative approach
|
||||
that does not require this future import, and states:
|
||||
|
||||
If this PEP is accepted, PEP 563 will be deprecated and eventually removed.
|
||||
|
||||
However, the PEP does not provide a detailed plan for this deprecation.
|
||||
|
||||
There is some previous discussion of this topic `on Discourse <https://discuss.python.org/t/pep-649-deferred-evaluation-of-annotations-tentatively-accepted/21331/44>`__
|
||||
(note that in the linked post I proposed something different from what is proposed here).
|
||||
|
||||
Specification
|
||||
-------------
|
||||
|
||||
We suggest the following deprecation plan:
|
||||
|
||||
- In Python 3.14, ``from __future__ import annotations`` will continue to work as it
|
||||
did before, converting annotations into strings.
|
||||
- Sometime after the last release that did not support :pep:`649` semantics (expected to be 3.13)
|
||||
reaches its end-of-life, ``from __future__ import annotations`` is deprecated. Compiling
|
||||
any code that uses the future import will emit a :py:exc:`DeprecationWarning`. This will
|
||||
happen no sooner than the first release after Python 3.13 reaches its end-of-life, but
|
||||
the community may decide to wait longer.
|
||||
- After at least two releases, the future import is removed, and annotations are always
|
||||
evaluated as per :pep:`649`. Code that continues to use the future import will raise
|
||||
a :py:exc:`SyntaxError`, similar to any other undefined future import.
|
||||
|
||||
Rejected alternatives
|
||||
---------------------
|
||||
|
||||
*Immediately make the future import a no-op*: We considered applying :pep:`649` semantics
|
||||
to all code in Python 3.14, making the future import a no-op. However, this would break
|
||||
code that works in 3.13 under the following set of conditions:
|
||||
|
||||
* ``__future__ import annotations`` is active
|
||||
* There are annotations that rely on forward references
|
||||
* Annotations are eagerly evaluated at import time, for example by a metaclass or
|
||||
class or function decorator. For example, this currently applies to the
|
||||
released version of ``typing_extensions.TypedDict``.
|
||||
|
||||
This is expected to be a common pattern, so we cannot afford to break such code during
|
||||
the upgrade from 3.13 to 3.14.
|
||||
|
||||
Such code would still break when the future import is eventually removed. However, this
|
||||
is many years in the future, giving affected decorators plenty of time to update their code.
|
||||
|
||||
*Immediately deprecate the future import*: Instead of waiting until Python 3.13 reaches
|
||||
its end-of-life, we could immediately start emitting warnings when the future import is
|
||||
used. However, many libraries are already using ``from __future__ import annotations`` as
|
||||
an elegant way to enable unrestricted forward references in their annotations. If we deprecate
|
||||
the future import immediately, it would be impossible for these libraries to use unrestricted
|
||||
forward references on all supported Python versions while avoiding deprecation warnings:
|
||||
unlike other features deprecated from the standard library, a ``__future__`` import must
|
||||
be the first statement in a given module, meaning it would be impossible to only
|
||||
conditionally import ``__future__.annotations`` on Python 3.13 and lower. (The necessary
|
||||
``sys.version_info`` check would count as a statement preceding the ``__future__`` import.)
|
||||
|
||||
*Keep the future import around forever*: We could also decide to keep the future import
|
||||
indefinitely. However, this would permanently bifurcate the behavior of the Python
|
||||
language. This is undesirable; the language should have only a single set of semantics,
|
||||
not two permanently different modes.
|
||||
|
||||
New ``annotations`` module
|
||||
==========================
|
||||
|
||||
:pep:`649` proposes to add tooling related to annotations to the :py:mod:`inspect`
|
||||
module. However, that module is rather large, has direct or indirect dependencies
|
||||
on at least 35 other standard library modules, and is so slow to import that other
|
||||
standard library modules are often discouraged from importing it. Furthermore, we
|
||||
anticipate adding more tools in addition to the :py:func:`inspect.get_annotations`
|
||||
function and the ``VALUE``, ``FORWARDREF``, and ``SOURCE`` formats.
|
||||
|
||||
A new standard library module provides a logical home for this functionality and
|
||||
also enables us to add more tooling that is useful for consumers of annotations.
|
||||
|
||||
Rationale
|
||||
---------
|
||||
|
||||
:pep:`649` indicates that :py:class:`!typing.ForwardRef` should be used to implement the
|
||||
``FORWARDREF`` format in :py:func:`inspect.get_annotations`. However, the existing implementation
|
||||
of :py:class:`!typing.ForwardRef` is intertwined with the rest of the :py:mod:`!typing` module,
|
||||
and it would not make sense to add :py:mod:`!typing`-specific behavior to the generic ``get_annotations()``
|
||||
function. Furthermore, :py:class:`!typing.ForwardRef` is a problematic
|
||||
class: it is public and documented, but the documentation lists no attributes or methods
|
||||
for it. Nonetheless, third-party libraries make use of some of its undocumented
|
||||
attributes. For instance, `Pydantic <https://github.com/pydantic/pydantic/blob/00ff77ed37589d924d3c10e0d5a48a7ef679a0d7/pydantic/v1/typing.py#L66>`__
|
||||
and `Typeguard <https://github.com/agronholm/typeguard/blob/016f8139f5a0a63147d68df9558cc5584cd2c49a/src/typeguard/_utils.py#L44>`__
|
||||
use the ``_evaluate`` method; `beartype <https://github.com/beartype/beartype/blob/0b4453f83c7ed4be054d8733aab8075e1478e166/beartype/_util/hint/pep/proposal/pep484585/utilpep484585ref.py#L210>`__
|
||||
and `pyanalyze <https://github.com/quora/pyanalyze/blob/9e401724f9d035cf138b72612834b6d5a00eb8e8/pyanalyze/annotations.py#L509>`__
|
||||
use the ``__forward_arg__`` attribute.
|
||||
|
||||
We replace the existing but poorly specified :py:class:`!typing.ForwardRef` with a new class,
|
||||
``annotations.ForwardRef``. It is designed to be mostly compatible with existing uses
|
||||
of the :py:class:`!typing.ForwardRef` class, but without the behaviors specific to the
|
||||
:py:mod:`!typing` module. For compatibility with existing users, we keep the private
|
||||
``_evaluate`` method, but mark it as deprecated. It delegates to a new public function in
|
||||
the :py:mod:`!typing` module, ``typing.evaluate_forward_ref``, that is designed to
|
||||
evaluate forward references in a way that is specific to type hints.
|
||||
|
||||
We add a function ``annotations.call_annotate_function`` as a helper for calling
|
||||
``__annotate__`` functions. This is a useful building block when implementing functionality
|
||||
that needs to partially evaluate annotations while a class is being constructed.
|
||||
For example, the implementation of :py:class:`typing.NamedTuple` needs to retrieve
|
||||
the annotations from a class namespace dictionary before the namedtuple class itself
|
||||
can be constructed, because the annotations determine what fields exist on the namedtuple.
|
||||
|
||||
Specification
|
||||
-------------
|
||||
|
||||
A new module, ``annotations``, is added to the standard library. Its aim is to
|
||||
provide tooling for introspecting and wrapping annotations.
|
||||
|
||||
The exact contents of the module are not yet specified. We will add support for
|
||||
:pep:`649` semantics to standard library functionality that uses annotations, such
|
||||
as :py:mod:`dataclasses` and :py:class:`typing.TypedDict`, and use the experience
|
||||
to inform the design of the new module.
|
||||
|
||||
The module will contain the following functionality:
|
||||
|
||||
* ``get_annotations()``: A function that returns the annotations of a function,
|
||||
module, or class. This will replace :py:func:`inspect.get_annotations`. The latter
|
||||
will delegate to the new function. It may eventually be deprecated, but to
|
||||
minimize disruption, we do not propose an immediate deprecation.
|
||||
* ``Format``: an enum that contains the possible formats of annotations. This will
|
||||
replace the ``VALUE``, ``FORWARDREF``, and ``SOURCE`` formats in :pep:`649`.
|
||||
PEP 649 proposed to make these values global members of the :py:mod:`inspect`
|
||||
module; we prefer to place them within an enum.
|
||||
* ``ForwardRef``: a class representing a forward reference; it may be returned by
|
||||
``get_annotations()`` when the format is ``FORWARDREF``. The existing class
|
||||
:py:class:`typing.ForwardRef` will become an alias of this class. Its members include:
|
||||
|
||||
* ``__forward_arg__``: the string argument of the forward reference
|
||||
* ``evaluate(globals=None, locals=None, type_params=None, owner=None)``: a method that attempts to evaluate
|
||||
the forward reference. The ``ForwardRef`` object may hold a reference to the
|
||||
globals and other namespaces of the object that it originated from. If so, these
|
||||
namespaces may be used to evaluate the forward reference. The *owner* argument
|
||||
may be the object that holds the original annotation, such as the class or module
|
||||
object; it is used to extract the globals and locals namespaces if these are not
|
||||
provided.
|
||||
* ``_evaluate()``, with the same interface as the existing ``ForwardRef._evaluate``
|
||||
method. It will be undocumented and immediately deprecated. It is provided for
|
||||
compatibility with existing users of ``typing.ForwardRef``.
|
||||
|
||||
* ``call_annotate_function(func: Callable, format: Format)``: a helper for calling
|
||||
an ``__annotate__`` function with a given format. If the function does not support
|
||||
this format, ``call_annotate_function()`` will set up a "fake globals" environment,
|
||||
as described in :pep:`649`, and use that environment to return the desired annotations
|
||||
format.
|
||||
* ``call_evaluate_function(func: Callable | None, format: Format)``: similar to
|
||||
``call_annotate_function``, but does not rely on the function returning an annotations
|
||||
dictionary. This is intended to be used for evaluating deferred attributes introduced by
|
||||
:pep:`695` and :pep:`696`; see below for details. *func* may be ``None``
|
||||
for convenience; if ``None`` is passed, the function also returns ``None``.
|
||||
|
||||
A new function is also added to the :py:mod:`!typing` module, ``typing.evaluate_forward_ref``.
|
||||
This function is a wrapper around the ``ForwardRef.evaluate`` method, but it performs
|
||||
additional work that is specific to type hints. For example, it recurses into complex
|
||||
types and evaluates additional forward references within these types.
|
||||
|
||||
Open issues
|
||||
-----------
|
||||
|
||||
What should this module be called? Some ideas:
|
||||
|
||||
- ``annotations``: The most obvious name, but it may cause confusion with the existing
|
||||
``from __future__ import annotations``. There is a PyPI package :pypi:`annotations`,
|
||||
but it had only a single release in 2015 and looks abandoned.
|
||||
- ``annotools``: Analogous to :py:mod:`itertools` and :py:mod:`functools`, but "anno" is a less
|
||||
obvious abbreviation than "iter" or "func". As of this writing, there
|
||||
is no PyPI package with this name.
|
||||
- ``annotationtools``: A more explicit version. There is a PyPI package
|
||||
:pypi:`annotationtools`, which had a release in 2023.
|
||||
- ``annotation_tools``: A variation of the above but without a PyPI conflict. However,
|
||||
no other public standard library module has an underscore in its name.
|
||||
- ``annotationslib``: Analogous to :py:mod:`tomllib`, :py:mod:`pathlib`, and :py:mod:`importlib`.
|
||||
There is no PyPI package with this name.
|
||||
|
||||
Rejected alternatives
|
||||
---------------------
|
||||
|
||||
*Add the functionality to the inspect module*: As described above, the
|
||||
:py:mod:`inspect` module is already quite large, and its import time is prohibitive
|
||||
for some use cases.
|
||||
|
||||
*Add the functionality to the typing module*: While annotations are mostly
|
||||
used for typing, they may also be used for other purposes. We prefer to keep a clean
|
||||
separation between functionality for introspecting annotations and functionality that
|
||||
is exclusively meant for type hints.
|
||||
|
||||
*Add the functionality to the types module*: The :py:mod:`types` module is
|
||||
meant for functionality related to *types*, and annotations can exist on functions
|
||||
and modules, not only on types.
|
||||
|
||||
*Develop this functionality in a third-party package*: The functionality in this new
|
||||
module will be pure Python code, and it is possible to implement a third-party package
|
||||
that provides the same functionality by interacting directly with ``__annotate__``
|
||||
functions generated by the interpreter. However, the functionality of the proposed new
|
||||
module will certainly be useful in the standard library itself (e.g., for implementing
|
||||
:py:mod:`dataclasses` and :py:class:`typing.NamedTuple`), so it makes sense to include
|
||||
it in the standard library.
|
||||
|
||||
*Add this functionality to a private module*: It would be possible to initially develop
|
||||
the module in a private standard library module (e.g., ``_annotations``), and publicize
|
||||
it only after we have gained more experience with the API. However, we already know
|
||||
that we will need parts of this module for the standard library itself (e.g., for
|
||||
implementing :py:mod:`!dataclasses` and :py:class:`!typing.NamedTuple`). Even if we make
|
||||
it private, the module will inevitably get used by third-party users. It is preferable
|
||||
to start with a clear, documented API from the beginning, to enable third-party users to
|
||||
support :pep:`649` semantics as thoroughly as the standard library. The module will
|
||||
immediately be used in other parts of the standard library, ensuring that it covers a
|
||||
reasonable set of use cases.
|
||||
|
||||
Behavior of the REPL
|
||||
====================
|
||||
|
||||
:pep:`649` specifies the following behavior of the interactive REPL:
|
||||
|
||||
For the sake of simplicity, in this case we forego delayed evaluation.
|
||||
Module-level annotations in the REPL shell will continue to work exactly
|
||||
as they do with “stock semantics”, evaluating immediately and setting the
|
||||
result directly inside the ``__annotations__`` dict.
|
||||
|
||||
There are several problems with this proposed behavior. It makes the REPL the
|
||||
only context where annotations are still evaluated immediately, which is
|
||||
confusing for users and complicates the language.
|
||||
|
||||
It also makes the implementation of the REPL more complex, as it needs to
|
||||
ensure that all statements are compiled in "interactive" mode, even if their
|
||||
output does not need to be displayed. (This matters if there are multiple
|
||||
statements in a single line evaluated by the REPL.)
|
||||
|
||||
Most importantly, this breaks some plausible use cases that inexperienced
|
||||
users could run into. A user might write the following in a file::
|
||||
|
||||
a: X | None = None
|
||||
class X: ...
|
||||
|
||||
Under :pep:`649` this would work fine: ``X`` is not yet defined when it is used
|
||||
in the annotation for ``a``, but the annotation is lazily evaluated. However,
|
||||
if a user were to paste this same code into the REPL and execute it line by
|
||||
line, it would throw a ``NameError``, because the name ``X`` is not yet defined.
|
||||
|
||||
This topic was previously discussed `on Discourse <https://discuss.python.org/t/pep-649-behavior-of-the-repl/54109>`__.
|
||||
|
||||
Specification
|
||||
-------------
|
||||
|
||||
We propose to treat the interactive console like any other module-level code, and
|
||||
make annotations lazily evaluated. This makes the language more consistent and
|
||||
avoids subtle behavior changes between modules and the REPL.
|
||||
|
||||
Because the REPL is evaluated line by line, we would generate a new ``__annotate__``
|
||||
function for every evaluated statement in the global scope that contains annotations. Whenever a line
|
||||
containing annotations is evaluated, the previous ``__annotate__`` function is
|
||||
lost:
|
||||
|
||||
.. code:: pycon
|
||||
|
||||
>>> x: int
|
||||
>>> __annotate__(1)
|
||||
{'x': <class 'int'>}
|
||||
>>> y: str
|
||||
>>> __annotate__(1)
|
||||
{'y': <class 'str'>}
|
||||
>>> z: doesntexist
|
||||
>>> __annotate__(1)
|
||||
Traceback (most recent call last):
|
||||
File "<python-input-5>", line 1, in <module>
|
||||
__annotate__(1)
|
||||
~~~~~~~~~~~~^^^
|
||||
File "<python-input-4>", line 1, in __annotate__
|
||||
z: doesntexist
|
||||
^^^^^^^^^^^
|
||||
NameError: name 'doesntexist' is not defined
|
||||
|
||||
There will be no ``__annotations__`` key in the global namespace of the REPL.
|
||||
In module namespaces, this key is created lazily when the ``__annotations__``
|
||||
descriptor of the module object is accessed, but in the REPL there is no such module
|
||||
object.
|
||||
|
||||
Classes and functions defined within the REPL will also work like any other classes,
|
||||
so evaluation of their annotations will be deferred. It is possible to access the
|
||||
``__annotations__`` and ``__annotate__`` attributes or use the ``annotations`` module
|
||||
to introspect the annotations.
|
||||
|
||||
Wrappers that provide ``__annotations__``
|
||||
=========================================
|
||||
|
||||
Several objects in the standard library and elsewhere provide annotations for their
|
||||
wrapped object. :pep:`649` does not specify how such wrappers should behave.
|
||||
|
||||
Specification
|
||||
-------------
|
||||
|
||||
Wrappers that provide annotations should be designed with the following goals
|
||||
in mind:
|
||||
|
||||
* Evaluation of ``__annotations__`` should be deferred for as long as possible,
|
||||
consistent with the behavior of built-in functions, classes, and modules.
|
||||
* Backward compatibility with the behavior prior to the implementation of :pep:`649`
|
||||
should be preserved.
|
||||
* The ``__annotate__`` and ``__annotations__`` attributes should both be supplied
|
||||
with semantics consistent to those of the wrapped object.
|
||||
|
||||
More specifically:
|
||||
|
||||
* :py:func:`functools.update_wrapper` (and therefore :py:func:`functools.wraps`)
|
||||
will copy only the ``__annotate__`` attribute
|
||||
from the wrapped object to the wrapper. The ``__annotations__`` descriptor on the
|
||||
wrapper function will use the copied ``__annotate__``.
|
||||
* The constructors for :py:func:`classmethod` and :py:func:`staticmethod` currently
|
||||
copy the ``__annotations__`` attribute from the wrapped object to the wrapper.
|
||||
They will instead have writable attributes for
|
||||
``__annotate__`` and ``__annotations__``. Reading these attributes will retrieve
|
||||
the corresponding attribute from the underlying callable and cache it in the wrapper's
|
||||
``__dict__``. Writing to these attributes will directly update the ``__dict__``,
|
||||
without affecting the wrapped callable.
|
||||
|
||||
Remove code flag for marking ``__annotate__`` functions
|
||||
=======================================================
|
||||
|
||||
:pep:`649` specifies:
|
||||
|
||||
This PEP assumes that
|
||||
third-party libraries may implement their own ``__annotate__``
|
||||
methods, and those functions would almost certainly work
|
||||
incorrectly when run in this "fake globals" environment.
|
||||
For that reason, this PEP allocates a flag on code objects,
|
||||
one of the unused bits in ``co_flags``, to mean "This code
|
||||
object can be run in a 'fake globals' environment." This
|
||||
makes the "fake globals" environment strictly opt-in, and
|
||||
it's expected that only ``__annotate__`` methods generated
|
||||
by the Python compiler will set it.
|
||||
|
||||
We have not found a need for this mechanism during our work to
|
||||
add :pep:`649` support to the standard library. While it is true
|
||||
that custom ``__annotate__`` functions may not work well with the
|
||||
"fake globals" environment, this technique is used only when the
|
||||
``__annotate__`` function raises :py:exc:`NotImplementedError` to
|
||||
signal that it does not support the requested format. However,
|
||||
manually implemented ``__annotate__`` functions are likely to support
|
||||
all three annotation formats; often, they will consist of a call to
|
||||
``annotations.call_annotate_function`` plus some transformation of the
|
||||
result.
|
||||
|
||||
In addition, the proposed mechanism couples the implementation with
|
||||
low-level details of the code object. The code object flags are
|
||||
CPython-specific and the documentation :py:ref:`explicitly warns <inspect-module-co-flags>`
|
||||
against relying on the values.
|
||||
|
||||
Specification
|
||||
-------------
|
||||
|
||||
The standard library will use the "fake globals" technique on any
|
||||
``__annotate__`` function that raises :py:exc:`NotImplementedError`
|
||||
when the requested format is not supported.
|
||||
|
||||
Third-party code that implements ``__annotate__`` functions should either
|
||||
support all three annotation formats, or be prepared to handle the
|
||||
"fake globals" environment. This should be mentioned in the data model
|
||||
documentation for ``__annotate__``.
|
||||
|
||||
Effect of setting ``__annotations__``
|
||||
=====================================
|
||||
|
||||
:pep:`649` specifies:
|
||||
|
||||
Setting ``o.__annotations__`` to a legal value
|
||||
automatically sets ``o.__annotate__`` to ``None``.
|
||||
|
||||
We would prefer to keep ``__annotate__`` unchanged when ``__annotations__``
|
||||
is written to. Conceptually, ``__annotate__`` provides the ground truth
|
||||
and ``__annotations__`` is merely a cache, and we shouldn't throw away the
|
||||
ground truth if the cache is modified.
|
||||
|
||||
The motivation for :pep:`649`'s behavior is to keep the two attributes in sync.
|
||||
However, this is impossible in general; if the ``__annotations__`` dictionary
|
||||
is modified in place, this will not be reflected in the ``__annotate__`` attribute.
|
||||
The overall mental model for this area will be simpler if setting ``__annotations__``
|
||||
has no effect on ``__annotate__``.
|
||||
|
||||
Specification
|
||||
-------------
|
||||
|
||||
The value of ``__annotate__`` is not changed when ``__annotations__`` is set.
|
||||
|
||||
Deferred evaluation of PEP 695 and 696 objects
|
||||
==============================================
|
||||
|
||||
Since :pep:`649` was written, Python 3.12 and 3.13 gained support for
|
||||
several new features that also use deferred evaluation, similar to the
|
||||
behavior this PEP proposes for annotations:
|
||||
|
||||
* The value of type aliases created through the :py:keyword:`type`
|
||||
statement (:pep:`695`)
|
||||
* The bound and constraints of :py:class:`typing.TypeVar` objects
|
||||
created through the syntax for generics (:pep:`695`)
|
||||
* The default value of :py:class:`typing.TypeVar`, :py:class:`ParamSpec`,
|
||||
and :py:class:`typing.TypeVarTuple` objects (:pep:`696`)
|
||||
|
||||
Currently, these objects use deferred evaluation, but there is no direct
|
||||
access to the function object used for deferred evaluation. To enable
|
||||
the same kind of introspection that is now possible for annotations, we propose
|
||||
to expose the internal function objects, allowing users to evaluate them
|
||||
using the FORWARDREF and SOURCE formats.
|
||||
|
||||
Specification
|
||||
-------------
|
||||
|
||||
We will add the following new attributes:
|
||||
|
||||
* ``evaluate_value`` on :py:class:`typing.TypeAliasType`
|
||||
* ``evaluate_bound``, ``evaluate_constraints``, and ``evaluate_default`` on :py:class:`typing.TypeVar`
|
||||
* ``evaluate_default`` on :py:class:`typing.ParamSpec`
|
||||
* ``evaluate_default`` on :py:class:`typing.TypeVarTuple`
|
||||
|
||||
Except for ``evaluate_value``, these attributes may be ``None`` if the object
|
||||
does not have a bound, constraints, or default. Otherwise, the attribute is a
|
||||
callable, similar to an ``__annotate__`` function, that takes a single integer
|
||||
argument and returns the evaluated value. Unlike ``__annotate__`` functions,
|
||||
these callables return a single value, not a dictionary of annotations.
|
||||
These attributes are read-only.
|
||||
|
||||
Usually, users would use these attributes in combinations with
|
||||
``annotations.call_evaluate_function``. For example, to get a ``TypeVar``'s bound
|
||||
in SOURCE format, one could write
|
||||
``annotations.call_evaluate_function(T.evaluate_bound, annotations.Format.SOURCE)``.
|
||||
|
||||
Miscellaneous implementation details
|
||||
====================================
|
||||
|
||||
:pep:`649` goes into considerable detail on some aspects of the implementation.
|
||||
To avoid confusion, we describe a few aspects where the current implementation
|
||||
differs from that described in the PEP. However, these details are not guaranteed
|
||||
to hold in the future, and they may change without notice in the future, unless
|
||||
they are documented in the language reference.
|
||||
|
||||
Supported operations on ``ForwardRef`` objects
|
||||
----------------------------------------------
|
||||
|
||||
The ``SOURCE`` format is implemented by the "stringizer" technique,
|
||||
where the globals dictionary of a function is augmented so that every
|
||||
lookup results in a special object that can be used to reconstruct the
|
||||
operations that are performed on the object.
|
||||
|
||||
:pep:`649` specifies:
|
||||
|
||||
In practice, the "stringizer" functionality will be implemented
|
||||
in the ``ForwardRef`` object currently defined in the
|
||||
``typing`` module. ``ForwardRef`` will be extended to
|
||||
implement all stringizer functionality; it will also be
|
||||
extended to support evaluating the string it contains,
|
||||
to produce the real value (assuming all symbols referenced
|
||||
are defined).
|
||||
|
||||
However, this is likely to lead to confusion in practice. An object
|
||||
that implements stringizer functionality must implement almost all
|
||||
special methods, including ``__getattr__`` and ``__eq__``, to return
|
||||
a new stringizer. Such an object is confusing to work with: all operations
|
||||
succeed, but they are likely to return different objects than the user
|
||||
expects.
|
||||
|
||||
The current implementation instead implements only a few useful methods
|
||||
on the ``ForwardRef`` class. During the evaluation of annotations,
|
||||
an instance of a private stringizer class is used instead of ``ForwardRef``.
|
||||
After evaluation completes, the implementation of the FORWARDREF format
|
||||
converts these internal objects into ``ForwardRef`` objects.
|
||||
|
||||
Backwards Compatibility
|
||||
=======================
|
||||
|
||||
:pep:`649` provides a thorough discussion of the backwards compatibility implications
|
||||
on existing code that uses either stock or :pep:`563` semantics.
|
||||
|
||||
However, there is another set of compatibility problems: new code that is written
|
||||
assuming :pep:`649` semantics, but uses existing tools that eagerly evaluate annotations.
|
||||
For example, consider a ``dataclass``-like class decorator ``@annotator`` that retrieves the annotated
|
||||
fields in the class it decorates, either by accessing ``__annotations__`` directly
|
||||
or by calling :py:func:`inspect.get_annotations`.
|
||||
|
||||
Once :pep:`649` is implemented, code like this will work fine::
|
||||
|
||||
class X:
|
||||
y: Y
|
||||
|
||||
class Y: pass
|
||||
|
||||
But this will not, unless ``@annotator`` is changed to use the new ``FORWARDREF``
|
||||
format::
|
||||
|
||||
@annotator
|
||||
class X:
|
||||
y: Y
|
||||
|
||||
class Y: pass
|
||||
|
||||
This is not strictly a backwards compatibility issue, since no previously working code
|
||||
would break; before :pep:`649`, this code would have raised ``NameError`` at runtime.
|
||||
In a sense, it is no different from any other new Python feature that needs
|
||||
to be supported by third-party libraries. Nevertheless, it is a serious issue for libraries
|
||||
that perform introspection, and it is important that we make it as easy as possible for
|
||||
libraries to support the new semantics in a straightforward, user-friendly way.
|
||||
|
||||
We will update those parts of the standard library that are affected by this problem,
|
||||
and we propose to add commonly useful functionality to the new ``annotations`` module,
|
||||
so third-party tools can use the same set of tools.
|
||||
|
||||
|
||||
Security Implications
|
||||
=====================
|
||||
|
||||
None.
|
||||
|
||||
|
||||
How to Teach This
|
||||
=================
|
||||
|
||||
The semantics of :pep:`649`, as modified by this PEP, should largely be intuitive for
|
||||
users who add annotations to their code. We eliminate the need for manually adding
|
||||
quotes around annotations that require forward references, a major source of confusion
|
||||
for users.
|
||||
|
||||
For advanced users who need to introspect annotations, the story becomes more complex.
|
||||
The documentation of the new ``annotations`` module will serve as a reference for users
|
||||
who need to interact programmatically with annotations.
|
||||
|
||||
|
||||
Reference Implementation
|
||||
========================
|
||||
|
||||
The in-progress PR `#119891 <https://github.com/python/cpython/pull/119891>`__
|
||||
implements much of this PEP.
|
||||
|
||||
Open Issues
|
||||
===========
|
||||
|
||||
We may discover additional areas where :pep:`649` needs clarification or amendment
|
||||
as we make progress on implementing it. Readers are encouraged to follow the
|
||||
`CPython issue <https://github.com/python/cpython/issues/119180>`__ tracking the
|
||||
implementation of the PEP and try out the draft implementation. Any feedback may
|
||||
be incorporated into future versions of this PEP.
|
||||
|
||||
Should dataclass field types use deferred evaluation?
|
||||
-----------------------------------------------------
|
||||
|
||||
The current draft implementation already supports deferred evaluation in dataclasses,
|
||||
so this works:
|
||||
|
||||
.. code:: pycon
|
||||
|
||||
>>> from dataclasses import dataclass
|
||||
>>> @dataclass
|
||||
... class D:
|
||||
... x: undefined
|
||||
...
|
||||
|
||||
However, the ``FORWARDREF`` format leaks into the field types of the dataclass:
|
||||
|
||||
.. code:: pycon
|
||||
|
||||
>>> fields(D)[0].type
|
||||
ForwardRef('undefined')
|
||||
|
||||
We could instead add deferred evaluation for the field type, similar to that outlined
|
||||
above for type alias values.
|
||||
|
||||
Accessing ``.type`` might throw an error:
|
||||
|
||||
.. code:: pycon
|
||||
|
||||
>>> @dataclass
|
||||
... class D:
|
||||
... x: undefined
|
||||
...
|
||||
>>> field = fields(D)[0]
|
||||
>>> field.type
|
||||
Traceback (most recent call last):
|
||||
File "<python-input-4>", line 1, in <module>
|
||||
field.type
|
||||
File ".../dataclasses.py", line 308, in type
|
||||
annos = self._annotate(annotations.Format.VALUE)
|
||||
File "<python-input-2>", line 3, in __annotate__
|
||||
x: undefined
|
||||
^^^^^^^^^
|
||||
NameError: name 'undefined' is not defined
|
||||
|
||||
But users could use ``annotations.call_evaluate_function`` to get the type in other formats:
|
||||
|
||||
.. code:: pycon
|
||||
|
||||
>>> annotations.call_evaluate_function(field.evaluate_type, annotations.Format.SOURCE)
|
||||
'undefined'
|
||||
>>> annotations.call_evaluate_function(field.evaluate_type, annotations.Format.FORWARDREF)
|
||||
ForwardRef('undefined')
|
||||
|
||||
Other variations are possible. For example, we could leave the ``type`` attribute unchanged,
|
||||
and only add the ``evaluate_type`` method. This avoids unpleasant surprises where accessing
|
||||
``.type`` may throw an exception.
|
||||
|
||||
Acknowledgments
|
||||
===============
|
||||
|
||||
First of all, I thank Larry Hastings for writing :pep:`649`. This PEP modifies some of his
|
||||
initial decisions, but the overall design is still his.
|
||||
|
||||
I thank Carl Meyer and Alex Waygood for feedback on early drafts of this PEP.
|
||||
|
||||
Appendix
|
||||
========
|
||||
|
||||
Which expressions can be stringified?
|
||||
-------------------------------------
|
||||
|
||||
:pep:`649` acknowledges that the stringifier cannot handle all expressions. Now that we
|
||||
have a draft implementation, we can be more precise about the expressions that can and
|
||||
cannot be handled. Below is a list of all expressions in the Python AST that can and
|
||||
cannot be recovered by the stringifier. The full list should probably not be added to
|
||||
the documentation, but creating it is a useful exercise.
|
||||
|
||||
First, the stringifier of course cannot recover any information that is not present in
|
||||
the compiled code, including comments, whitespace, parenthesization, and operations that
|
||||
get simplified by the AST optimizer.
|
||||
|
||||
Second, the stringifier can intercept almost all operations that involve names looked
|
||||
up in some scope, but it cannot intercept operations that operate fully on constants.
|
||||
As a corollary, this also means it is not safe to request the ``SOURCE`` format on
|
||||
untrusted code: Python is powerful enough that it is possible to achieve arbitrary
|
||||
code execution even with no access to any globals or builtins. For example:
|
||||
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> def f(x: (1).__class__.__base__.__subclasses__()[-1].__init__.__builtins__["print"]("Hello world")): pass
|
||||
...
|
||||
>>> annotations.get_annotations(f, format=annotations.Format.SOURCE)
|
||||
Hello world
|
||||
{'x': 'None'}
|
||||
|
||||
(This particular example worked for me on the current implementation of a draft of this PEP;
|
||||
the exact code may not keep working in the future.)
|
||||
|
||||
The following are supported (sometimes with caveats):
|
||||
|
||||
* ``BinOp``
|
||||
* ``UnaryOp``
|
||||
|
||||
* ``Invert`` (``~``), ``UAdd`` (``+``), and ``USub`` (``-``) are supported
|
||||
* ``Not`` (``not``) is not supported
|
||||
|
||||
* ``Dict`` (except when using ``**`` unpacking)
|
||||
* ``Set``
|
||||
* ``Compare``
|
||||
|
||||
* ``Eq`` and ``NotEq`` are supported
|
||||
* ``Lt``, ``LtE``, ``Gt``, and ``GtE`` are supported, but the operand may be flipped
|
||||
* ``Is``, ``IsNot``, ``In``, and ``NotIn`` are not supported
|
||||
|
||||
* ``Call`` (except when using ``**`` unpacking)
|
||||
* ``Constant`` (though not the exact representation of the constant; for example, escape
|
||||
sequences in strings are lost; hexadecimal numbers are converted to decimal)
|
||||
* ``Attribute`` (assuming the value is not a constant)
|
||||
* ``Subscript`` (assuming the value is not a constant)
|
||||
* ``Starred`` (``*`` unpacking)
|
||||
* ``Name``
|
||||
* ``List``
|
||||
* ``Tuple``
|
||||
* ``Slice``
|
||||
|
||||
The following are unsupported, but throw an informative error when encountered by the
|
||||
stringifier:
|
||||
|
||||
* ``FormattedValue`` (f-strings; error is not detected if conversion specifiers like ``!r``
|
||||
are used)
|
||||
* ``JoinedStr`` (f-strings)
|
||||
|
||||
The following are unsupported and result in incorrect output:
|
||||
|
||||
* ``BoolOp`` (``and`` and ``or``)
|
||||
* ``IfExp``
|
||||
* ``Lambda``
|
||||
* ``ListComp``
|
||||
* ``SetComp``
|
||||
* ``DictComp``
|
||||
* ``GeneratorExp``
|
||||
|
||||
The following are disallowed in annotation scopes and therefore not relevant:
|
||||
|
||||
* ``NamedExpr`` (``:=``)
|
||||
* ``Await``
|
||||
* ``Yield``
|
||||
* ``YieldFrom``
|
||||
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
This document is placed in the public domain or under the
|
||||
CC0-1.0-Universal license, whichever is more permissive.
|
||||
Loading…
Reference in New Issue