iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

PEP 692: Abandon the syntax change proposal (#2941) 

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
This commit is contained in:
Franek Magiera 2022-12-29 22:56:28 +01:00 committed by GitHub
parent d1cfb37937
commit e3010cb7d2
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 111 additions and 178 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

289
pep-0692.rst
View File

@ -22,8 +22,7 @@ arguments specified by them are of the same type. However, that behaviour can
be very limiting. Therefore, in this PEP we propose a new way to enable more
precise ``**kwargs`` typing. The new approach revolves around using
``TypedDict`` to type ``**kwargs`` that comprise keyword arguments of different
types. It also involves introducing a grammar change and a new dunder
``__typing_unpack__``.
types.
.. _pep-692-motivation:
@ -70,6 +69,8 @@ that are usually used as defaults to indicate no user input, such as ``None``,
can be passed in by a user and should result in a valid, non-default behavior.
For example, this issue `came up <httpxIssue1384_>`__ in the popular ``httpx`` library.
.. _pep-692-rationale:
Rationale
=========
@ -94,17 +95,28 @@ For instance::
means that each keyword argument in ``foo`` is itself a ``Movie`` dictionary
that has a ``name`` key with a string type value and a ``year`` key with an
integer type value. Therefore, in order to support specifying ``kwargs`` type
as a ``TypedDict`` without breaking current behaviour, a new syntax has to be
introduced.
as a ``TypedDict`` without breaking current behaviour, a new construct has to
be introduced.
To support this use case, we propose reusing ``Unpack`` which
was initially introduced in :pep:`646`. There are several reasons for doing so:
* Its name is quite suitable and intuitive for the ``**kwargs`` typing use case
as our intention is to "unpack" the keywords arguments from the supplied
``TypedDict``.
* The current way of typing ``*args`` would be extended to ``**kwargs``
and those are supposed to behave similarly.
* There would be no need to introduce any new special forms.
* The use of ``Unpack`` for the purposes described in this PEP does not
interfere with the use cases described in :pep:`646`.
Specification
=============
To support the aforementioned use case we propose to use the double asterisk
syntax inside of the type annotation. The required grammar change is discussed
in more detail in section `Grammar Changes`_. Continuing the previous example::
With ``Unpack`` we introduce a new way of annotating ``**kwargs``.
Continuing the previous example::
def foo(**kwargs: **Movie) -> None: ...
def foo(**kwargs: Unpack[Movie]) -> None: ...
would mean that the ``**kwargs`` comprise two keyword arguments specified by
``Movie`` (i.e. a ``name`` keyword of type ``str`` and a ``year`` keyword of
@ -115,10 +127,10 @@ type ``int``). This indicates that the function should be called as follows::
foo(**kwargs) # OK!
foo(name="The Meaning of Life", year=1983) # OK!
Inside the function itself, the type checkers should treat
the ``kwargs`` parameter as a ``TypedDict``::
When ``Unpack`` is used, type checkers treat ``kwargs`` inside the
function body as a ``TypedDict``::
def foo(**kwargs: **Movie) -> None:
def foo(**kwargs: Unpack[Movie]) -> None:
assert_type(kwargs, Movie) # OK!
@ -129,12 +141,12 @@ sections relates to type checker errors.
Function calls with standard dictionaries
-----------------------------------------
Calling a function that has ``**kwargs`` typed using the ``**kwargs: **Movie``
syntax with a dictionary of type ``dict[str, object]`` must generate a type
checker error. On the other hand, the behaviour for functions using standard,
untyped dictionaries can depend on the type checker. For example::
Passing a dictionary of type ``dict[str, object]`` as a ``**kwargs`` argument
to a function that has ``**kwargs`` annotated with ``Unpack`` must generate a
type checker error. On the other hand, the behaviour for functions using
standard, untyped dictionaries can depend on the type checker. For example::
def foo(**kwargs: **Movie) -> None: ...
def foo(**kwargs: Unpack[Movie]) -> None: ...
movie: dict[str, object] = {"name": "Life of Brian", "year": 1979}
foo(**movie) # WRONG! Movie is of type dict[str, object]
@ -154,14 +166,14 @@ name is a standard parameter, an error should be reported by type checkers.
If the duplicate name is a positional-only parameter, no errors should be
generated. For example::
def foo(name, **kwargs: **Movie) -> None: ... # WRONG! "name" will
# always bind to the
# first parameter.
def foo(name, **kwargs: Unpack[Movie]) -> None: ... # WRONG! "name" will
# always bind to the
# first parameter.
def foo(name, /, **kwargs: **Movie) -> None: ... # OK! "name" is a
# positional parameter,
# so **kwargs can contain
# a "name" keyword.
def foo(name, /, **kwargs: Unpack[Movie]) -> None: ... # OK! "name" is a
# positional-only parameter,
# so **kwargs can contain
# a "name" keyword.
Required and non-required keys
------------------------------
@ -184,14 +196,14 @@ caller, then an error must be reported by type checkers.
Assignment
----------
Assignments of a function typed with the ``**kwargs: **Movie`` construct and
Assignments of a function typed with ``**kwargs: Unpack[Movie]`` and
another callable type should pass type checking only if they are compatible.
This can happen for the scenarios described below.
Source and destination contain ``**kwargs``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Both destination and source functions have a ``**kwargs: **TypedDict``
Both destination and source functions have a ``**kwargs: Unpack[TypedDict]``
parameter and the destination function's ``TypedDict`` is assignable to the
source function's ``TypedDict`` and the rest of the parameters are
compatible::
@ -202,8 +214,8 @@ compatible::
class Dog(Animal):
breed: str
def accept_animal(**kwargs: **Animal): ...
def accept_dog(**kwargs: **Dog): ...
def accept_animal(**kwargs: Unpack[Animal]): ...
def accept_dog(**kwargs: Unpack[Dog]): ...
accept_dog = accept_animal # OK! Expression of type Dog can be
# assigned to a variable of type Animal.
@ -217,7 +229,7 @@ Source contains ``**kwargs`` and destination doesn't
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The destination callable doesn't contain ``**kwargs``, the source callable
contains ``**kwargs: **TypedDict`` and the destination function's keyword
contains ``**kwargs: Unpack[TypedDict]`` and the destination function's keyword
arguments are assignable to the corresponding keys in source function's
``TypedDict``. Moreover, not required keys should correspond to optional
function arguments, whereas required keys should correspond to required
@ -229,7 +241,7 @@ Continuing the previous example::
string: str
number: NotRequired[int]
def src(**kwargs: **Example): ...
def src(**kwargs: Unpack[Example]): ...
def dest(*, animal: Dog, string: str, number: int = ...): ...
dest = src # OK!
@ -246,13 +258,13 @@ only::
# keyword arguments.
The reverse situation where the destination callable contains
``**kwargs: **TypedDict`` and the source callable doesn't contain
``**kwargs: Unpack[TypedDict]`` and the source callable doesn't contain
``**kwargs`` should be disallowed. This is because, we cannot be sure that
additional keyword arguments are not being passed in when an instance of a
subclass had been assigned to a variable with a base class type and then
unpacked in the destination callable invocation::
def dest(**Animal): ...
def dest(**kwargs: Unpack[Animal]): ...
def src(name: str): ...
dog: Dog = {"name": "Daisy", "breed": "Labrador"}
@ -267,18 +279,18 @@ between ``TypedDict``\s is based on structural subtyping.
Source contains untyped ``**kwargs``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The destination callable contains ``**kwargs: **TypedDict`` and the source
callable contains untyped ``**kwargs``::
The destination callable contains ``**kwargs: Unpack[TypedDict]`` and the
source callable contains untyped ``**kwargs``::
def src(**kwargs): ...
def dest(**kwargs: **Movie): ...
def dest(**kwargs: Unpack[Movie]): ...
dest = src # OK!
Source contains traditionally typed ``**kwargs: T``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The destination callable contains ``**kwargs: **TypedDict``, the source
The destination callable contains ``**kwargs: Unpack[TypedDict]``, the source
callable contains traditionally typed ``**kwargs: T`` and each of the
destination function ``TypedDict``'s fields is assignable to a variable of
type ``T``::
@ -296,14 +308,14 @@ type ``T``::
car: Car
moto: Motorcycle
def dest(**kwargs: **Vehicles): ...
def dest(**kwargs: Unpack[Vehicles]): ...
def src(**kwargs: Vehicle): ...
dest = src # OK!
On the other hand, if the destination callable contains either untyped or
traditionally typed ``**kwargs: T`` and the source callable is typed using
``**kwargs: **TypedDict`` then an error should be generated, because
``**kwargs: Unpack[TypedDict]`` then an error should be generated, because
traditionally typed ``**kwargs`` aren't checked for keyword names.
To summarize, function parameters should behave contravariantly and function
@ -328,16 +340,16 @@ consider the following example::
dog: Dog = {"name": "Daisy", "breed": "Labrador"}
animal: Animal = dog
def foo(**kwargs: **Animal):
def foo(**kwargs: Unpack[Animal]):
print(kwargs["name"].capitalize())
def bar(**kwargs: **Animal):
def bar(**kwargs: Unpack[Animal]):
takes_name(**kwargs)
def baz(animal: Animal):
takes_name(**animal)
def spam(**kwargs: **Animal):
def spam(**kwargs: Unpack[Animal]):
baz(kwargs)
foo(**animal) # OK! foo only expects and uses keywords of 'Animal'.
@ -366,14 +378,35 @@ In cases similar to the ``bar`` function above the problem could be worked
around by explicitly dereferencing desired fields and using them as arguments
to perform the function call::
def bar(**kwargs: **Animal):
def bar(**kwargs: Unpack[Animal]):
name = kwargs["name"]
takes_name(name)
Using ``Unpack`` with types other than ``TypedDict``
----------------------------------------------------
As described in the :ref:`Rationale <pep-692-rationale>` section,
``TypedDict`` is the most natural candidate for typing ``**kwargs``.
Therefore, in the context of typing ``**kwargs``, using ``Unpack`` with types
other than ``TypedDict`` should not be allowed and type checkers should
generate errors in such cases.
Changes to ``Unpack``
---------------------
Currently using ``Unpack`` in the context of
typing is interchangeable with using the asterisk syntax::
>>> Unpack[Movie]
*<class '__main__.Movie'>
Therefore, in order to be compatible with the new use case, ``Unpack``'s
``repr`` should be changed to simply ``Unpack[T]``.
.. _pep-692-intended-usage:
Intended Usage
--------------
==============
The intended use cases for this proposal are described in the
:ref:`pep-692-motivation` section. In summary, more precise ``**kwargs`` typing
can bring benefits to already existing codebases that decided to use
@ -390,8 +423,8 @@ PEP. For example, when writing new code if all the keyword arguments are
required or have default values then writing everything explicitly is better
than using ``**kwargs`` and a ``TypedDict``::
def foo(name: str, year: int): ... # Preferred way.
def foo(**kwargs: **Movie): ...
def foo(name: str, year: int): ... # Preferred way.
def foo(**kwargs: Unpack[Movie]): ...
Similarly, when type hinting third party libraries via stubs it is again better
to state the function signature explicitly - this is the only way to type such
@ -401,7 +434,7 @@ standard function parameters may be treated as keyword only::
def foo(name, year): ... # Function in a third party library.
def foo(**Movie): ... # Function signature in a stub file.
def foo(Unpack[Movie]): ... # Function signature in a stub file.
foo("Life of Brian", 1979) # This would be now failing type
# checking but is fine.
@ -418,128 +451,13 @@ explicitly as::
Also, for the benefit of IDEs and documentation pages, functions that are part
of the public API should prefer explicit keyword parameters whenever possible.
Grammar Changes
===============
This PEP requires a grammar change so that the double asterisk syntax is
allowed for ``**kwargs`` annotations. The proposed change is to extend the
``kwds`` rule in `the grammar <https://docs.python.org/3/reference/grammar.html>`__
as follows:
Before:
.. code-block:: peg
kwds: '**' param_no_default
After:
.. code-block:: peg
kwds:
| '**' param_no_default_double_star_annotation
| '**' param_no_default
param_no_default_double_star_annotation:
| param_double_star_annotation ','? &')'
param_double_star_annotation: NAME double_star_annotation
double_star_annotation: ':' double_star_expression
double_star_expression: '**' expression
A new AST node needs to be created so that type checkers can differentiate the
semantics of the new syntax from the existing one, which indicates that all
``**kwargs`` should be of the same type. Then, whenever the new syntax is
used, type checkers will be able to take into account that ``**kwargs`` should
be unpacked. The proposition is to add a new ``DoubleStarred`` AST node. Then,
an AST node for the function defined as::
def foo(**kwargs: **Movie): ...
should look as below::
FunctionDef(
name='foo',
args=arguments(
posonlyargs=[],
args=[],
kwonlyargs=[],
kw_defaults=[],
kwarg=arg(
arg='kwargs',
annotation=DoubleStarred(
value=Name(id='Movie', ctx=Load()),
ctx=Load())),
defaults=[]),
body=[
Expr(
value=Constant(value=Ellipsis))],
decorator_list=[])
The runtime annotations should be consistent with the AST. Continuing the
previous example::
>>> def foo(**kwargs: **Movie): ...
...
>>> foo.__annotations__
{'kwargs': Unpack[Movie]}
To accomplish this, we propose a new dunder called ``__typing_unpack__``.
The double asterisk syntax should result in a call to the ``__typing_unpack__``
special method on an object it was used on. This means that at runtime,
``def foo(**kwargs: **T): ...`` is equivalent to
``def foo(**kwargs: type(T).__typing_unpack__(T)): ...``.
``TypedDict`` is the only type in the standard library that is expected to
implement ``__typing_unpack__``, which should return ``Unpack[self]``. The
motivation for reusing :pep:`646`'s ``Unpack`` is described in the
:ref:`Backwards Compatibility <pep-692-backwards-compatibility>` section.
It is worth pointing out that currently using ``Unpack`` in the context of
typing is interchangeable with using the asterisk syntax::
>>> Unpack[Movie]
*<class '__main__.Movie'>
Therefore, in order to be compatible with the new usecase, ``Unpack``'s
``repr`` should be changed to simply ``Unpack[T]``.
.. _pep-692-backwards-compatibility:
Backwards Compatibility
-----------------------
Using the double asterisk syntax for annotating ``**kwargs`` would be available
only in new versions of Python. :pep:`646` dealt with the similar problem and
its authors introduced a new type operator ``Unpack``. For the purposes of this
PEP, the proposition is to reuse ``Unpack`` for more precise ``**kwargs``
typing. For example::
def foo(**kwargs: Unpack[Movie]) -> None: ...
There are several reasons for reusing :pep:`646`'s ``Unpack``. Firstly, the
name is quite suitable and intuitive for the ``**kwargs`` typing use case as
the keywords arguments are "unpacked" from the ``TypedDict``. Secondly, there
would be no need to introduce any new special forms. Lastly, the use of
``Unpack`` for the purposes described in this PEP does not interfere with the
use cases described in :pep:`646`.
Alternatives
------------
Instead of making the grammar change, ``Unpack`` could be the only way to
annotate ``**kwargs`` of different types. However, introducing the double
asterisk syntax has two advantages. Namely, it is more concise and more
intuitive than using ``Unpack``.
How to Teach This
=================
This PEP could be linked in the ``typing`` module's documentation. Moreover, a
new section on using ``Unpack`` as well as the new double asterisk syntax could
be added to the aforementioned docs. Similar sections could be also added to
the `mypy documentation <https://mypy.readthedocs.io/>`_ and the
new section on using ``Unpack`` could be added to the aforementioned docs.
Similar sections could be also added to the
`mypy documentation <https://mypy.readthedocs.io/>`_ and the
`typing RTD documentation <https://typing.readthedocs.io/>`_.
Reference Implementation
@ -553,9 +471,6 @@ The `mypy type checker <https://github.com/python/mypy>`_ already
`provides provisional support <pyrightProvisionalImplementation_>`__
for `this feature <pyrightIssue3002_>`__.
A proof-of-concept implementation of the CPython `grammar changes`_ described in
this PEP is `available on GitHub <cpythonGrammarChangePoc_>`__.
Rejected Ideas
==============
@ -575,28 +490,28 @@ can result in an error::
TypedDictUnion = Movie | Book
def foo(**kwargs: **TypedDictUnion) -> None: ... # WRONG! Unsupported use
# of a union of
# TypedDicts to type
# **kwargs
def foo(**kwargs: Unpack[TypedDictUnion]) -> None: ... # WRONG! Unsupported use
# of a union of
# TypedDicts to type
# **kwargs
Instead, a function that expects a union of ``TypedDict``\s can be
overloaded::
@overload
def foo(**kwargs: **Movie): ...
def foo(**kwargs: Unpack[Movie]): ...
@overload
def foo(**kwargs: **Book): ...
def foo(**kwargs: Unpack[Book]): ...
Changing the meaning of ``**kwargs`` annotations
------------------------------------------------
One way to achieve the purpose of this PEP without any grammar
change would be to change the meaning of ``**kwargs`` annotations,
so that the annotations would apply to the entire ``**kwargs`` dict,
not to individual elements. For consistency, we would have to make an
analogous change to ``*args`` annotations.
One way to achieve the purpose of this PEP would be to change the
meaning of ``**kwargs`` annotations, so that the annotations would
apply to the entire ``**kwargs`` dict, not to individual elements.
For consistency, we would have to make an analogous change to ``*args``
annotations.
This idea was discussed in a meeting of the typing community, and the
consensus was that the change would not be worth the cost. There is no
@ -604,6 +519,25 @@ clear migration path, the current meaning of ``*args`` and ``**kwargs``
annotations is well-established in the ecosystem, and type checkers
would have to introduce new errors for code that is currently legal.
Introducing a new syntax
------------------------
In the previous versions of this PEP, using a double asterisk syntax was
proposed to support more precise ``**kwargs`` typing. Using this syntax,
functions could be annotated as follows::
def foo(**kwargs: **Movie): ...
Which would have the same meaning as::
def foo(**kwargs: Unpack[Movie]): ...
This greatly increased the scope of the PEP, as it would require a grammar
change and adding a new dunder for the ``Unpack`` special form. At the same
the justification for introducing a new syntax was not strong enough and
became a blocker for the whole PEP. Therefore, we decided to abandon the idea
of introducing a new syntax as a part of this PEP and may propose it again in a
separate one.
References
==========
@ -612,7 +546,6 @@ References
.. _mypyIssue4441: https://github.com/python/mypy/issues/4441
.. _pyrightIssue3002: https://github.com/microsoft/pyright/issues/3002
.. _pyrightProvisionalImplementation: https://github.com/microsoft/pyright/commit/5bee749eb171979e3f526cd8e5bf66b00593378a
.. _cpythonGrammarChangePoc: https://github.com/python/cpython/compare/main...franekmagiera:annotate-kwargs
Copyright
=========