PEP 692: Abandon the syntax change proposal (#2941)

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
This commit is contained in:
Franek Magiera 2022-12-29 22:56:28 +01:00 committed by GitHub
parent d1cfb37937
commit e3010cb7d2
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 111 additions and 178 deletions

View File

@ -22,8 +22,7 @@ arguments specified by them are of the same type. However, that behaviour can
be very limiting. Therefore, in this PEP we propose a new way to enable more be very limiting. Therefore, in this PEP we propose a new way to enable more
precise ``**kwargs`` typing. The new approach revolves around using precise ``**kwargs`` typing. The new approach revolves around using
``TypedDict`` to type ``**kwargs`` that comprise keyword arguments of different ``TypedDict`` to type ``**kwargs`` that comprise keyword arguments of different
types. It also involves introducing a grammar change and a new dunder types.
``__typing_unpack__``.
.. _pep-692-motivation: .. _pep-692-motivation:
@ -70,6 +69,8 @@ that are usually used as defaults to indicate no user input, such as ``None``,
can be passed in by a user and should result in a valid, non-default behavior. can be passed in by a user and should result in a valid, non-default behavior.
For example, this issue `came up <httpxIssue1384_>`__ in the popular ``httpx`` library. For example, this issue `came up <httpxIssue1384_>`__ in the popular ``httpx`` library.
.. _pep-692-rationale:
Rationale Rationale
========= =========
@ -94,17 +95,28 @@ For instance::
means that each keyword argument in ``foo`` is itself a ``Movie`` dictionary means that each keyword argument in ``foo`` is itself a ``Movie`` dictionary
that has a ``name`` key with a string type value and a ``year`` key with an that has a ``name`` key with a string type value and a ``year`` key with an
integer type value. Therefore, in order to support specifying ``kwargs`` type integer type value. Therefore, in order to support specifying ``kwargs`` type
as a ``TypedDict`` without breaking current behaviour, a new syntax has to be as a ``TypedDict`` without breaking current behaviour, a new construct has to
introduced. be introduced.
To support this use case, we propose reusing ``Unpack`` which
was initially introduced in :pep:`646`. There are several reasons for doing so:
* Its name is quite suitable and intuitive for the ``**kwargs`` typing use case
as our intention is to "unpack" the keywords arguments from the supplied
``TypedDict``.
* The current way of typing ``*args`` would be extended to ``**kwargs``
and those are supposed to behave similarly.
* There would be no need to introduce any new special forms.
* The use of ``Unpack`` for the purposes described in this PEP does not
interfere with the use cases described in :pep:`646`.
Specification Specification
============= =============
To support the aforementioned use case we propose to use the double asterisk With ``Unpack`` we introduce a new way of annotating ``**kwargs``.
syntax inside of the type annotation. The required grammar change is discussed Continuing the previous example::
in more detail in section `Grammar Changes`_. Continuing the previous example::
def foo(**kwargs: **Movie) -> None: ... def foo(**kwargs: Unpack[Movie]) -> None: ...
would mean that the ``**kwargs`` comprise two keyword arguments specified by would mean that the ``**kwargs`` comprise two keyword arguments specified by
``Movie`` (i.e. a ``name`` keyword of type ``str`` and a ``year`` keyword of ``Movie`` (i.e. a ``name`` keyword of type ``str`` and a ``year`` keyword of
@ -115,10 +127,10 @@ type ``int``). This indicates that the function should be called as follows::
foo(**kwargs) # OK! foo(**kwargs) # OK!
foo(name="The Meaning of Life", year=1983) # OK! foo(name="The Meaning of Life", year=1983) # OK!
Inside the function itself, the type checkers should treat When ``Unpack`` is used, type checkers treat ``kwargs`` inside the
the ``kwargs`` parameter as a ``TypedDict``:: function body as a ``TypedDict``::
def foo(**kwargs: **Movie) -> None: def foo(**kwargs: Unpack[Movie]) -> None:
assert_type(kwargs, Movie) # OK! assert_type(kwargs, Movie) # OK!
@ -129,12 +141,12 @@ sections relates to type checker errors.
Function calls with standard dictionaries Function calls with standard dictionaries
----------------------------------------- -----------------------------------------
Calling a function that has ``**kwargs`` typed using the ``**kwargs: **Movie`` Passing a dictionary of type ``dict[str, object]`` as a ``**kwargs`` argument
syntax with a dictionary of type ``dict[str, object]`` must generate a type to a function that has ``**kwargs`` annotated with ``Unpack`` must generate a
checker error. On the other hand, the behaviour for functions using standard, type checker error. On the other hand, the behaviour for functions using
untyped dictionaries can depend on the type checker. For example:: standard, untyped dictionaries can depend on the type checker. For example::
def foo(**kwargs: **Movie) -> None: ... def foo(**kwargs: Unpack[Movie]) -> None: ...
movie: dict[str, object] = {"name": "Life of Brian", "year": 1979} movie: dict[str, object] = {"name": "Life of Brian", "year": 1979}
foo(**movie) # WRONG! Movie is of type dict[str, object] foo(**movie) # WRONG! Movie is of type dict[str, object]
@ -154,14 +166,14 @@ name is a standard parameter, an error should be reported by type checkers.
If the duplicate name is a positional-only parameter, no errors should be If the duplicate name is a positional-only parameter, no errors should be
generated. For example:: generated. For example::
def foo(name, **kwargs: **Movie) -> None: ... # WRONG! "name" will def foo(name, **kwargs: Unpack[Movie]) -> None: ... # WRONG! "name" will
# always bind to the # always bind to the
# first parameter. # first parameter.
def foo(name, /, **kwargs: **Movie) -> None: ... # OK! "name" is a def foo(name, /, **kwargs: Unpack[Movie]) -> None: ... # OK! "name" is a
# positional parameter, # positional-only parameter,
# so **kwargs can contain # so **kwargs can contain
# a "name" keyword. # a "name" keyword.
Required and non-required keys Required and non-required keys
------------------------------ ------------------------------
@ -184,14 +196,14 @@ caller, then an error must be reported by type checkers.
Assignment Assignment
---------- ----------
Assignments of a function typed with the ``**kwargs: **Movie`` construct and Assignments of a function typed with ``**kwargs: Unpack[Movie]`` and
another callable type should pass type checking only if they are compatible. another callable type should pass type checking only if they are compatible.
This can happen for the scenarios described below. This can happen for the scenarios described below.
Source and destination contain ``**kwargs`` Source and destination contain ``**kwargs``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Both destination and source functions have a ``**kwargs: **TypedDict`` Both destination and source functions have a ``**kwargs: Unpack[TypedDict]``
parameter and the destination function's ``TypedDict`` is assignable to the parameter and the destination function's ``TypedDict`` is assignable to the
source function's ``TypedDict`` and the rest of the parameters are source function's ``TypedDict`` and the rest of the parameters are
compatible:: compatible::
@ -202,8 +214,8 @@ compatible::
class Dog(Animal): class Dog(Animal):
breed: str breed: str
def accept_animal(**kwargs: **Animal): ... def accept_animal(**kwargs: Unpack[Animal]): ...
def accept_dog(**kwargs: **Dog): ... def accept_dog(**kwargs: Unpack[Dog]): ...
accept_dog = accept_animal # OK! Expression of type Dog can be accept_dog = accept_animal # OK! Expression of type Dog can be
# assigned to a variable of type Animal. # assigned to a variable of type Animal.
@ -217,7 +229,7 @@ Source contains ``**kwargs`` and destination doesn't
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The destination callable doesn't contain ``**kwargs``, the source callable The destination callable doesn't contain ``**kwargs``, the source callable
contains ``**kwargs: **TypedDict`` and the destination function's keyword contains ``**kwargs: Unpack[TypedDict]`` and the destination function's keyword
arguments are assignable to the corresponding keys in source function's arguments are assignable to the corresponding keys in source function's
``TypedDict``. Moreover, not required keys should correspond to optional ``TypedDict``. Moreover, not required keys should correspond to optional
function arguments, whereas required keys should correspond to required function arguments, whereas required keys should correspond to required
@ -229,7 +241,7 @@ Continuing the previous example::
string: str string: str
number: NotRequired[int] number: NotRequired[int]
def src(**kwargs: **Example): ... def src(**kwargs: Unpack[Example]): ...
def dest(*, animal: Dog, string: str, number: int = ...): ... def dest(*, animal: Dog, string: str, number: int = ...): ...
dest = src # OK! dest = src # OK!
@ -246,13 +258,13 @@ only::
# keyword arguments. # keyword arguments.
The reverse situation where the destination callable contains The reverse situation where the destination callable contains
``**kwargs: **TypedDict`` and the source callable doesn't contain ``**kwargs: Unpack[TypedDict]`` and the source callable doesn't contain
``**kwargs`` should be disallowed. This is because, we cannot be sure that ``**kwargs`` should be disallowed. This is because, we cannot be sure that
additional keyword arguments are not being passed in when an instance of a additional keyword arguments are not being passed in when an instance of a
subclass had been assigned to a variable with a base class type and then subclass had been assigned to a variable with a base class type and then
unpacked in the destination callable invocation:: unpacked in the destination callable invocation::
def dest(**Animal): ... def dest(**kwargs: Unpack[Animal]): ...
def src(name: str): ... def src(name: str): ...
dog: Dog = {"name": "Daisy", "breed": "Labrador"} dog: Dog = {"name": "Daisy", "breed": "Labrador"}
@ -267,18 +279,18 @@ between ``TypedDict``\s is based on structural subtyping.
Source contains untyped ``**kwargs`` Source contains untyped ``**kwargs``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The destination callable contains ``**kwargs: **TypedDict`` and the source The destination callable contains ``**kwargs: Unpack[TypedDict]`` and the
callable contains untyped ``**kwargs``:: source callable contains untyped ``**kwargs``::
def src(**kwargs): ... def src(**kwargs): ...
def dest(**kwargs: **Movie): ... def dest(**kwargs: Unpack[Movie]): ...
dest = src # OK! dest = src # OK!
Source contains traditionally typed ``**kwargs: T`` Source contains traditionally typed ``**kwargs: T``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The destination callable contains ``**kwargs: **TypedDict``, the source The destination callable contains ``**kwargs: Unpack[TypedDict]``, the source
callable contains traditionally typed ``**kwargs: T`` and each of the callable contains traditionally typed ``**kwargs: T`` and each of the
destination function ``TypedDict``'s fields is assignable to a variable of destination function ``TypedDict``'s fields is assignable to a variable of
type ``T``:: type ``T``::
@ -296,14 +308,14 @@ type ``T``::
car: Car car: Car
moto: Motorcycle moto: Motorcycle
def dest(**kwargs: **Vehicles): ... def dest(**kwargs: Unpack[Vehicles]): ...
def src(**kwargs: Vehicle): ... def src(**kwargs: Vehicle): ...
dest = src # OK! dest = src # OK!
On the other hand, if the destination callable contains either untyped or On the other hand, if the destination callable contains either untyped or
traditionally typed ``**kwargs: T`` and the source callable is typed using traditionally typed ``**kwargs: T`` and the source callable is typed using
``**kwargs: **TypedDict`` then an error should be generated, because ``**kwargs: Unpack[TypedDict]`` then an error should be generated, because
traditionally typed ``**kwargs`` aren't checked for keyword names. traditionally typed ``**kwargs`` aren't checked for keyword names.
To summarize, function parameters should behave contravariantly and function To summarize, function parameters should behave contravariantly and function
@ -328,16 +340,16 @@ consider the following example::
dog: Dog = {"name": "Daisy", "breed": "Labrador"} dog: Dog = {"name": "Daisy", "breed": "Labrador"}
animal: Animal = dog animal: Animal = dog
def foo(**kwargs: **Animal): def foo(**kwargs: Unpack[Animal]):
print(kwargs["name"].capitalize()) print(kwargs["name"].capitalize())
def bar(**kwargs: **Animal): def bar(**kwargs: Unpack[Animal]):
takes_name(**kwargs) takes_name(**kwargs)
def baz(animal: Animal): def baz(animal: Animal):
takes_name(**animal) takes_name(**animal)
def spam(**kwargs: **Animal): def spam(**kwargs: Unpack[Animal]):
baz(kwargs) baz(kwargs)
foo(**animal) # OK! foo only expects and uses keywords of 'Animal'. foo(**animal) # OK! foo only expects and uses keywords of 'Animal'.
@ -366,14 +378,35 @@ In cases similar to the ``bar`` function above the problem could be worked
around by explicitly dereferencing desired fields and using them as arguments around by explicitly dereferencing desired fields and using them as arguments
to perform the function call:: to perform the function call::
def bar(**kwargs: **Animal): def bar(**kwargs: Unpack[Animal]):
name = kwargs["name"] name = kwargs["name"]
takes_name(name) takes_name(name)
Using ``Unpack`` with types other than ``TypedDict``
----------------------------------------------------
As described in the :ref:`Rationale <pep-692-rationale>` section,
``TypedDict`` is the most natural candidate for typing ``**kwargs``.
Therefore, in the context of typing ``**kwargs``, using ``Unpack`` with types
other than ``TypedDict`` should not be allowed and type checkers should
generate errors in such cases.
Changes to ``Unpack``
---------------------
Currently using ``Unpack`` in the context of
typing is interchangeable with using the asterisk syntax::
>>> Unpack[Movie]
*<class '__main__.Movie'>
Therefore, in order to be compatible with the new use case, ``Unpack``'s
``repr`` should be changed to simply ``Unpack[T]``.
.. _pep-692-intended-usage: .. _pep-692-intended-usage:
Intended Usage Intended Usage
-------------- ==============
The intended use cases for this proposal are described in the The intended use cases for this proposal are described in the
:ref:`pep-692-motivation` section. In summary, more precise ``**kwargs`` typing :ref:`pep-692-motivation` section. In summary, more precise ``**kwargs`` typing
can bring benefits to already existing codebases that decided to use can bring benefits to already existing codebases that decided to use
@ -390,8 +423,8 @@ PEP. For example, when writing new code if all the keyword arguments are
required or have default values then writing everything explicitly is better required or have default values then writing everything explicitly is better
than using ``**kwargs`` and a ``TypedDict``:: than using ``**kwargs`` and a ``TypedDict``::
def foo(name: str, year: int): ... # Preferred way. def foo(name: str, year: int): ... # Preferred way.
def foo(**kwargs: **Movie): ... def foo(**kwargs: Unpack[Movie]): ...
Similarly, when type hinting third party libraries via stubs it is again better Similarly, when type hinting third party libraries via stubs it is again better
to state the function signature explicitly - this is the only way to type such to state the function signature explicitly - this is the only way to type such
@ -401,7 +434,7 @@ standard function parameters may be treated as keyword only::
def foo(name, year): ... # Function in a third party library. def foo(name, year): ... # Function in a third party library.
def foo(**Movie): ... # Function signature in a stub file. def foo(Unpack[Movie]): ... # Function signature in a stub file.
foo("Life of Brian", 1979) # This would be now failing type foo("Life of Brian", 1979) # This would be now failing type
# checking but is fine. # checking but is fine.
@ -418,128 +451,13 @@ explicitly as::
Also, for the benefit of IDEs and documentation pages, functions that are part Also, for the benefit of IDEs and documentation pages, functions that are part
of the public API should prefer explicit keyword parameters whenever possible. of the public API should prefer explicit keyword parameters whenever possible.
Grammar Changes
===============
This PEP requires a grammar change so that the double asterisk syntax is
allowed for ``**kwargs`` annotations. The proposed change is to extend the
``kwds`` rule in `the grammar <https://docs.python.org/3/reference/grammar.html>`__
as follows:
Before:
.. code-block:: peg
kwds: '**' param_no_default
After:
.. code-block:: peg
kwds:
| '**' param_no_default_double_star_annotation
| '**' param_no_default
param_no_default_double_star_annotation:
| param_double_star_annotation ','? &')'
param_double_star_annotation: NAME double_star_annotation
double_star_annotation: ':' double_star_expression
double_star_expression: '**' expression
A new AST node needs to be created so that type checkers can differentiate the
semantics of the new syntax from the existing one, which indicates that all
``**kwargs`` should be of the same type. Then, whenever the new syntax is
used, type checkers will be able to take into account that ``**kwargs`` should
be unpacked. The proposition is to add a new ``DoubleStarred`` AST node. Then,
an AST node for the function defined as::
def foo(**kwargs: **Movie): ...
should look as below::
FunctionDef(
name='foo',
args=arguments(
posonlyargs=[],
args=[],
kwonlyargs=[],
kw_defaults=[],
kwarg=arg(
arg='kwargs',
annotation=DoubleStarred(
value=Name(id='Movie', ctx=Load()),
ctx=Load())),
defaults=[]),
body=[
Expr(
value=Constant(value=Ellipsis))],
decorator_list=[])
The runtime annotations should be consistent with the AST. Continuing the
previous example::
>>> def foo(**kwargs: **Movie): ...
...
>>> foo.__annotations__
{'kwargs': Unpack[Movie]}
To accomplish this, we propose a new dunder called ``__typing_unpack__``.
The double asterisk syntax should result in a call to the ``__typing_unpack__``
special method on an object it was used on. This means that at runtime,
``def foo(**kwargs: **T): ...`` is equivalent to
``def foo(**kwargs: type(T).__typing_unpack__(T)): ...``.
``TypedDict`` is the only type in the standard library that is expected to
implement ``__typing_unpack__``, which should return ``Unpack[self]``. The
motivation for reusing :pep:`646`'s ``Unpack`` is described in the
:ref:`Backwards Compatibility <pep-692-backwards-compatibility>` section.
It is worth pointing out that currently using ``Unpack`` in the context of
typing is interchangeable with using the asterisk syntax::
>>> Unpack[Movie]
*<class '__main__.Movie'>
Therefore, in order to be compatible with the new usecase, ``Unpack``'s
``repr`` should be changed to simply ``Unpack[T]``.
.. _pep-692-backwards-compatibility:
Backwards Compatibility
-----------------------
Using the double asterisk syntax for annotating ``**kwargs`` would be available
only in new versions of Python. :pep:`646` dealt with the similar problem and
its authors introduced a new type operator ``Unpack``. For the purposes of this
PEP, the proposition is to reuse ``Unpack`` for more precise ``**kwargs``
typing. For example::
def foo(**kwargs: Unpack[Movie]) -> None: ...
There are several reasons for reusing :pep:`646`'s ``Unpack``. Firstly, the
name is quite suitable and intuitive for the ``**kwargs`` typing use case as
the keywords arguments are "unpacked" from the ``TypedDict``. Secondly, there
would be no need to introduce any new special forms. Lastly, the use of
``Unpack`` for the purposes described in this PEP does not interfere with the
use cases described in :pep:`646`.
Alternatives
------------
Instead of making the grammar change, ``Unpack`` could be the only way to
annotate ``**kwargs`` of different types. However, introducing the double
asterisk syntax has two advantages. Namely, it is more concise and more
intuitive than using ``Unpack``.
How to Teach This How to Teach This
================= =================
This PEP could be linked in the ``typing`` module's documentation. Moreover, a This PEP could be linked in the ``typing`` module's documentation. Moreover, a
new section on using ``Unpack`` as well as the new double asterisk syntax could new section on using ``Unpack`` could be added to the aforementioned docs.
be added to the aforementioned docs. Similar sections could be also added to Similar sections could be also added to the
the `mypy documentation <https://mypy.readthedocs.io/>`_ and the `mypy documentation <https://mypy.readthedocs.io/>`_ and the
`typing RTD documentation <https://typing.readthedocs.io/>`_. `typing RTD documentation <https://typing.readthedocs.io/>`_.
Reference Implementation Reference Implementation
@ -553,9 +471,6 @@ The `mypy type checker <https://github.com/python/mypy>`_ already
`provides provisional support <pyrightProvisionalImplementation_>`__ `provides provisional support <pyrightProvisionalImplementation_>`__
for `this feature <pyrightIssue3002_>`__. for `this feature <pyrightIssue3002_>`__.
A proof-of-concept implementation of the CPython `grammar changes`_ described in
this PEP is `available on GitHub <cpythonGrammarChangePoc_>`__.
Rejected Ideas Rejected Ideas
============== ==============
@ -575,28 +490,28 @@ can result in an error::
TypedDictUnion = Movie | Book TypedDictUnion = Movie | Book
def foo(**kwargs: **TypedDictUnion) -> None: ... # WRONG! Unsupported use def foo(**kwargs: Unpack[TypedDictUnion]) -> None: ... # WRONG! Unsupported use
# of a union of # of a union of
# TypedDicts to type # TypedDicts to type
# **kwargs # **kwargs
Instead, a function that expects a union of ``TypedDict``\s can be Instead, a function that expects a union of ``TypedDict``\s can be
overloaded:: overloaded::
@overload @overload
def foo(**kwargs: **Movie): ... def foo(**kwargs: Unpack[Movie]): ...
@overload @overload
def foo(**kwargs: **Book): ... def foo(**kwargs: Unpack[Book]): ...
Changing the meaning of ``**kwargs`` annotations Changing the meaning of ``**kwargs`` annotations
------------------------------------------------ ------------------------------------------------
One way to achieve the purpose of this PEP without any grammar One way to achieve the purpose of this PEP would be to change the
change would be to change the meaning of ``**kwargs`` annotations, meaning of ``**kwargs`` annotations, so that the annotations would
so that the annotations would apply to the entire ``**kwargs`` dict, apply to the entire ``**kwargs`` dict, not to individual elements.
not to individual elements. For consistency, we would have to make an For consistency, we would have to make an analogous change to ``*args``
analogous change to ``*args`` annotations. annotations.
This idea was discussed in a meeting of the typing community, and the This idea was discussed in a meeting of the typing community, and the
consensus was that the change would not be worth the cost. There is no consensus was that the change would not be worth the cost. There is no
@ -604,6 +519,25 @@ clear migration path, the current meaning of ``*args`` and ``**kwargs``
annotations is well-established in the ecosystem, and type checkers annotations is well-established in the ecosystem, and type checkers
would have to introduce new errors for code that is currently legal. would have to introduce new errors for code that is currently legal.
Introducing a new syntax
------------------------
In the previous versions of this PEP, using a double asterisk syntax was
proposed to support more precise ``**kwargs`` typing. Using this syntax,
functions could be annotated as follows::
def foo(**kwargs: **Movie): ...
Which would have the same meaning as::
def foo(**kwargs: Unpack[Movie]): ...
This greatly increased the scope of the PEP, as it would require a grammar
change and adding a new dunder for the ``Unpack`` special form. At the same
the justification for introducing a new syntax was not strong enough and
became a blocker for the whole PEP. Therefore, we decided to abandon the idea
of introducing a new syntax as a part of this PEP and may propose it again in a
separate one.
References References
========== ==========
@ -612,7 +546,6 @@ References
.. _mypyIssue4441: https://github.com/python/mypy/issues/4441 .. _mypyIssue4441: https://github.com/python/mypy/issues/4441
.. _pyrightIssue3002: https://github.com/microsoft/pyright/issues/3002 .. _pyrightIssue3002: https://github.com/microsoft/pyright/issues/3002
.. _pyrightProvisionalImplementation: https://github.com/microsoft/pyright/commit/5bee749eb171979e3f526cd8e5bf66b00593378a .. _pyrightProvisionalImplementation: https://github.com/microsoft/pyright/commit/5bee749eb171979e3f526cd8e5bf66b00593378a
.. _cpythonGrammarChangePoc: https://github.com/python/cpython/compare/main...franekmagiera:annotate-kwargs
Copyright Copyright
========= =========