iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

PEP 612: Updates from typing-sig discussions (#1424)

This commit is contained in:
Mark Mendoza 2020-06-25 20:47:44 -07:00 committed by GitHub
parent b617b14c01
commit d32dc02bde
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 551 additions and 179 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

730
pep-0612.rst
View File

@ -17,13 +17,13 @@ Abstract
-------- --------
There currently are two ways to specify the type of a callable, the There currently are two ways to specify the type of a callable, the
``Callable[[T1, T2], TReturn]`` syntax defined in `PEP 484 ``Callable[[int, str], bool]`` syntax defined in `PEP 484
<https://www.python.org/dev/peps/pep-0484>`_\ , and callback protocols from `PEP <https://www.python.org/dev/peps/pep-0484>`_\ , and callback protocols from `PEP
544 <https://www.python.org/dev/peps/pep-0544/#callback-protocols>`_. Neither of 544 <https://www.python.org/dev/peps/pep-0544/#callback-protocols>`_. Neither of
these support forwarding the parameter types of one callable over to another these support forwarding the parameter types of one callable over to another
callable, making it difficult to annotate function decorators. This PEP proposes callable, making it difficult to annotate function decorators. This PEP proposes
``typing.ParameterSpecification``\ , a new kind of type variable, to support ``typing.ParamSpec`` and ``typing.type_variable_operators.Concatenate`` to
expressing these kinds of relationships. support expressing these kinds of relationships.
Motivation Motivation
---------- ----------
@ -35,91 +35,130 @@ tools to annotate the following common decorator pattern satisfactorily:
from typing import Awaitable, Callable, TypeVar from typing import Awaitable, Callable, TypeVar
TReturn = TypeVar("TReturn") R = TypeVar("R")
def add_logging( def add_logging(f: Callable[..., R]) -> Callable[..., Awaitable[R]]:
f: Callable[..., TReturn] async def inner(*args: object, **kwargs: object) -> R:
) -> Callable[..., Awaitable[TReturn]]:
async def inner(*args: object, **kwargs: object) -> TReturn:
await log_to_database() await log_to_database()
return f(*args, **kwargs) return f(*args, **kwargs)
return inner return inner
@add_logging @add_logging
def foo(x: int, y: str) -> int: def takes_int_str(x: int, y: str) -> int:
return x + 7 return x + 7
await foo(1, "A") await takes_int_str(1, "A")
await foo("B", 2) # fails at runtime await takes_int_str("B", 2) # fails at runtime
``add_logging``\ , a decorator which logs before each entry into the decorated ``add_logging``\ , a decorator which logs before each entry into the decorated
function, is an instance of the Python idiom of one function passing all function, is an instance of the Python idiom of one function passing all
arguments given to it over to another function through the combination of the arguments given to it over to another function. This is done through the
``*args`` and ``**kwargs`` features in both parameters and in arguments. When combination of the ``*args`` and ``**kwargs`` features in both parameters and in
one defines a function (like ``inner``\ ) that takes ``(*args, **kwargs)`` and arguments. When one defines a function (like ``inner``\ ) that takes ``(*args,
goes on to call another function with ``(*args, **kwargs)``\ , the wrapping **kwargs)`` and goes on to call another function with ``(*args, **kwargs)``\
function can only be safely called in all of the ways that the wrapped function , the wrapping function can only be safely called in all of the ways that the
could be safely called. To type this decorator, we’d like to be able to place wrapped function could be safely called. To type this decorator, we’d like to be
a dependency between the parameters of the callable ``f`` and the parameters of able to place a dependency between the parameters of the callable ``f`` and the
the returned function. `PEP 484 <https://www.python.org/dev/peps/pep-0484>`_ parameters of the returned function. `PEP 484
supports dependencies between single types, as in ``def append(l: <https://www.python.org/dev/peps/pep-0484>`_ supports dependencies between
typing.List[T], e: T) -> typing.List[T]: ...``\ , but there is no existing way single types, as in ``def append(l: typing.List[T], e: T) -> typing.List[T]:
to do so with a complicated entity like the parameters one could pass to ...``\ , but there is no existing way to do so with a complicated entity like
a function. the parameters of a function.
Due to the limitations of the status quo, the ``add_logging`` example will type Due to the limitations of the status quo, the ``add_logging`` example will type
check but will fail at runtime. ``inner`` will pass the string “B” into ``foo``\ check but will fail at runtime. ``inner`` will pass the string “B” into
, which will try to add 7 to it, triggering a type error. This was not caught ``takes_int_str``\, which will try to add 7 to it, triggering a type error.
by the type checker because the decorated ``foo`` was given the type This was not caught by the type checker because the decorated ``takes_int_str``
``Callable[..., Awaitable[int]]`` which is specified to do no validation on its was given the type ``Callable[..., Awaitable[int]]`` (an ellipsis in place of
arguments. parameter types is specified to mean that we do no validation on arguments).
Without the ability to define dependencies between the parameters of different Without the ability to define dependencies between the parameters of different
callable types, there is no way, at present, to make ``add_logging`` compatible callable types, there is no way, at present, to make ``add_logging`` compatible
with all functions, while still preserving the enforcement of the parameters of with all functions, while still preserving the enforcement of the parameters of
the decorated function. the decorated function.
With the addition of the ``ParameterSpecification`` variables proposed by this With the addition of the ``ParamSpec`` variables proposed by this
PEP, we can rewrite the previous example in a way that keeps the flexibility of PEP, we can rewrite the previous example in a way that keeps the flexibility of
the decorator and the parameter enforcement of the decorated function. the decorator and the parameter enforcement of the decorated function.
.. code-block:: .. code-block::
from typing import Awaitable, Callable, ParameterSpecification, TypeVar from typing import Awaitable, Callable, ParamSpec, TypeVar
Ps = ParameterSpecification("Ps") P = ParamSpec("P")
R = TypeVar("R") R = TypeVar("R")
def add_logging(f: Callable[Ps, R]) -> Callable[Ps, Awaitable[R]]: def add_logging(f: Callable[P, R]) -> Callable[P, Awaitable[R]]:
async def inner(*args: Ps.args, **kwargs: Ps.kwargs) -> R: async def inner(*args: P.args, **kwargs: P.kwargs) -> R:
await log_to_database() await log_to_database()
return f(*args, **kwargs) return f(*args, **kwargs)
return inner return inner
@add_logging @add_logging
def foo(x: int, y: str) -> int: def takes_int_str(x: int, y: str) -> int:
return x + 7 return x + 7
await foo(1, "A") await takes_int_str(1, "A") # Accepted
await foo("B", 2) # Incompatible parameter type: await takes_int_str("B", 2) # Correctly rejected by the type checker
# Expected `int` for 1st anonymous parameter to call `foo`
# but got `str` Another common decorator pattern that has previously been impossible to type is
the practice of adding or removing arguments from the decorated function. For
example:
.. code-block::
class Request:
...
def with_request(f: Callable[..., R]) -> Callable[..., R]:
def inner(*args: object, **kwargs: object) -> R:
return f(Request(), *args, **kwargs)
return inner
@with_request
def takes_int_str(request: Request, x: int, y: str) -> int:
# use request
return x + 7
takes_int_str(1, "A")
takes_int_str("B", 2) # fails at runtime
With the addition of the ``Concatenate`` operator from this PEP, we can even
type this more complex decorator.
.. code-block::
from typing.type_variable_operators import Concatenate
def with_request(f: Callable[Concatenate[Request, P], R]) -> Callable[P, R]:
def inner(*args: P.args, **kwargs: P.kwargs) -> R:
return f(Request(), *args, **kwargs)
return inner
@with_request
def takes_int_str(request: Request, x: int, y: str) -> int:
# use request
return x + 7
takes_int_str(1, "A") # Accepted
takes_int_str("B", 2) # Correctly rejected by the type checker
Specification Specification
------------- -------------
Declarations ParamSpec Declarations
^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^
A parameter specification variable is defined in a similar manner to a normal A parameter specification variable is defined in a similar manner to how a
``typing.TypeVar``. normal type variable is defined with ``typing.TypeVar``.
.. code-block:: .. code-block::
from typing import ParameterSpecification from typing import ParamSpec
TParams = ParameterSpecification("TParams") # Accepted P = ParamSpec("P") # Accepted
TParams = ParameterSpecification("WrongName") # Rejected P = ParamSpec("WrongName") # Rejected because P =/= WrongName
The runtime should accept ``bound``\ s and ``covariant`` and ``contravariant`` The runtime should accept ``bound``\ s and ``covariant`` and ``contravariant``
arguments in the declaration just as ``typing.TypeVar`` does, but for now we arguments in the declaration just as ``typing.TypeVar`` does, but for now we
@ -128,46 +167,100 @@ will defer the standardization of the semantics of those options to a later PEP.
Valid use locations Valid use locations
^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^
A declared ``ParameterSpecification`` can only be used in the place of the list Previously only a list of parameter arguments (``[A, B, C]``) or an ellipsis
of types in the declaration of a ``Callable`` type, or a user defined class (signifying "undefined parameters") were acceptable as the first "argument" to
which is generic in a ``ParameterSpecification`` variable (i.e., ``MyClass`` in ``typing.Callable`` . We now augment that with two new options: a parameter
the following example). specification variable (``Callable[P, int]``\ ) or a concatenation on a
parameter specification variable (``Callable[Concatenate[int, P], int]``\ ).
.. code-block:: .. code-block::
def foo( callable ::= Callable "[" parameters_expression, type_expression "]"
x: typing.Callable[TParams, int]
) -> typing.Callable[TParams, str]: # Accepted parameters_expression ::=
... | "..."
def foo( | "[" [ type_expression ("," type_expression)\* ] "]"
x: MyClass[TParams, int] | parameter_specification_variable
) -> typing.Callable[TParams, str]: # Accepted | concatenate "["
... type_expression ("," type_expression)\* ","
def foo(x: TParams) -> TParams: ... # Rejected parameter_specification_variable
def foo(x: typing.List[TParams]) -> None: ... # Rejected "]"
def foo(x: typing.Callable[[int, str], TParams]) -> None: ... # Rejected
where ``parameter_specification_variable`` is a ``typing.ParamSpec`` variable,
declared in the manner as defined above, and ``concatenate`` is
``typing.type_variable_operators.Concatenate``.
As before, ``parameters_expression``\ s by themselves are not acceptable in
places where a type is expected
.. code-block::
def foo(x: P) -> P: ... # Rejected
def foo(x: Concatenate[int, P]) -> int: ... # Rejected
def foo(x: typing.List[P]) -> None: ... # Rejected
def foo(x: Callable[[int, str], P]) -> None: ... # Rejected
User-Defined Generic Classes
````````````````````````````
Just as defining a class as inheriting from ``Generic[T]`` makes a class generic
for a single parameter (when ``T`` is a ``TypeVar``\ ), defining a class as
inheriting from ``Generic[P]`` makes a class generic on
``parameters_expression``\ s (when ``P`` is a ``ParamSpec``).
.. code-block::
T = TypeVar("T")
S = TypeVar("S")
P_2 = ParamSpec("P_2")
class X(Generic[T, P]):
...
def f(x: X[int, P_2]) -> str: ... # Accepted
def f(x: X[int, Concatenate[int, P_2]]) -> str: ... # Accepted
def f(x: X[int, [int, bool]]) -> str: ... # Accepted
def f(x: X[int, ...]) -> str: ... # Accepted
def f(x: X[int, int]) -> str: ... # Rejected
By the rules defined above, spelling an concrete instance of a class generic
with respect to only a single ``ParamSpec`` would require unsightly double
brackets. For aesthetic purposes we allow these to be omitted.
.. code-block::
class Z(Generic[P]):
...
def f(x: Z[[int, str, bool]]) -> str: ... # Accepted
def f(x: Z[int, str, bool]) -> str: ... # Equivalent
Semantics Semantics
^^^^^^^^^ ^^^^^^^^^
The inference rules for the return type of a function invocation whose signature The inference rules for the return type of a function invocation whose signature
contains a ``ParameterSpecification`` variable are analogous to those around contains a ``ParamSpec`` variable are analogous to those around
evaluating ones with ``TypeVar``\ s. evaluating ones with ``TypeVar``\ s.
.. code-block:: .. code-block::
def foo( def changes_return_type_to_str(x: Callable[P, int]) -> Callable[P, str]: ...
x: typing.Callable[TParams, int]
) -> typing.Callable[TParams, str]: ... def returns_int(a: str, b: bool) -> int: ...
def bar(a: str, b: bool) -> int: ...
f = foo(bar) # f should be inferred to have the same signature as bar, f = changes_return_type_to_str(returns_int) # f should have the type:
# but returning str # (a: str, b: bool) -> str
f("A", True) # Accepted
f(a="A", b=True) # Accepted f("A", True) # Accepted
f("A", "A") # Rejected f(a="A", b=True) # Accepted
f("A", "A") # Rejected
expects_str(f("A", True)) # Accepted
expects_int(f("A", True)) # Rejected
Just as with traditional ``TypeVars``\ , a user may include the same Just as with traditional ``TypeVars``\ , a user may include the same
``ParameterSpecification`` multiple times in the arguments of the same function, ``ParamSpec`` multiple times in the arguments of the same function,
to indicate a dependency between multiple arguments. In these cases a type to indicate a dependency between multiple arguments. In these cases a type
checker may choose to solve to a common behavioral supertype (i.e. a set of checker may choose to solve to a common behavioral supertype (i.e. a set of
parameters for which all of the valid calls are valid in both of the subtypes), parameters for which all of the valid calls are valid in both of the subtypes),
@ -175,122 +268,282 @@ but is not obligated to do so.
.. code-block:: .. code-block::
def foo( P = ParamSpec("P")
x: typing.Callable[TParams, int], y: typing.Callable[TParams, int]
) -> typing.Callable[TParams, bool]: ... def foo(x: Callable[P, int], y: Callable[P, int]) -> Callable[P, bool]: ...
def x_int_y_str(x: int, y: str) -> int: ... def x_int_y_str(x: int, y: str) -> int: ...
def y_int_x_str(y: int, x: str) -> int: ... def y_int_x_str(y: int, x: str) -> int: ...
foo(x_int_y_str, x_int_y_str) # Must return (x: int, y: str) -> int
foo(x_int_y_str, y_int_x_str) # Could return (__a: int, __b: str) -> int foo(x_int_y_str, x_int_y_str) # Should return (x: int, y: str) -> bool
# This works because both callables have types
# that are behavioral subtypes of foo(x_int_y_str, y_int_x_str) # Could return (__a: int, __b: str) -> bool
# Callable[[int, str], int] # This works because both callables have types
# that are behavioral subtypes of
# Callable[[int, str], object]
def keyword_only_x(*, x: int) -> int: ... def keyword_only_x(*, x: int) -> int: ...
def keyword_only_y(*, y: int) -> int: ... def keyword_only_y(*, y: int) -> int: ...
foo(keyword_only_x, keyword_only_y) # Must be rejected foo(keyword_only_x, keyword_only_y) # Rejected
Use in ``Generic`` Classes The semantics of ``Concatenate[X, Y, P]`` are that it represents the parameters
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ represented by ``P`` with two positional-only parameters prepended. This means
that we can use it to represent higher order functions that add, remove or
Just as with normal ``TypeVar``\ s, ``ParameterSpecification``\ s can be used to transform a finite number of parameters of a callable.
make generic classes as well as generic functions. These are able to be
mixed with normal ``TypeVar``\ s. This also work with
protocols in the same manner.
The components of a ``ParameterSpecification``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A ``ParameterSpecification`` captures both positional and keyword accessible
parameters, but there unfortunately is no object in the runtime that captures
both of these together. Instead, we are forced to separate them into ``*args``
and ``**kwargs``\ , respectively. This means we need to be able to split apart
a single ``ParameterSpecification`` into these two components, and then bring
them back together into a call. To do this, we introduce ``TParams.args`` to
represent the tuple of positional arguments in a given call and
``TParams.kwargs`` to represent the corresponding ``Mapping`` of keywords to
values. These operators can only be used together, as the annotated types for
``*args`` and ``**kwargs`` .
.. code-block:: .. code-block::
class G(Generic[TParams]): def bar(x: int, *args: bool) -> int: ...
def foo(
*args: TParams.args, **kwargs: TParams.kwargs
) -> int: # Accepted
...
def bar(
*args: TParams.kwargs, **kwargs: TParams.args
) -> int: # Rejected
...
def baz(*args: TParams.args) -> int: ... # Rejected
stored_arguments: TParams.args # Rejected
def bap(x: TParams.args) -> int: ... # Rejected
def bop(
*args: List[TParams.args], **kwargs: TParams.kwargs
) -> int: # Rejected
...
Because the default kind of parameter in Python (\ ``(x: int)``\ ) may be def add(x: Callable[P, int]) -> Callable[Concatenate[str, P], bool]: ...
addressed both positionally and through its name, two valid invocations of
a ``(*args: TParams.args, **kwargs: TParams.kwargs)`` function may give add(bar) # Should return (__a: str, x: int, *args: bool) -> bool
different partitions of the same set of parameters. Therefore we need to make
sure that these special types are only brought into the world together, and are def remove(x: Callable[Concatenate[int, P], int]) -> Callable[P, bool]: ...
used together, so that our usage is valid for all possible partitions.
remove(bar) # Should return (*args: bool) -> bool
def transform(
x: Callable[Concatenate[int, P], int]
) -> Callable[Concatenate[str, P], bool]: ...
transform(bar) # Should return (__a: str, *args: bool) -> bool
This also means that while any function that returns an ``R`` can satisfy
``typing.Callable[P, R]``, only functions that can be called positionally in
their first position with a ``X`` can satisfy
``typing.Callable[Concatenate[X, P], R]``.
.. code-block::
def expects_int_first(x: Callable[Concatenate[int, P], int]) -> None: ...
@expects_int_first # Rejected
def one(x: str) -> int: ...
@expects_int_first # Rejected
def two(*, x: int) -> int: ...
@expects_int_first # Rejected
def three(**kwargs: int) -> int: ...
@expects_int_first # Accepted
def four(*args: int) -> int: ...
There are still some classes of decorators still not supported with these
features:
* those that add/remove/change a **variable** number of parameters (for
example, ``functools.partial`` will remain untypable even after this PEP)
* those that add/remove/change keyword-only parameters (See
`Concatenating Keyword Parameters`_ for more details).
The components of a ``ParamSpec``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A ``ParamSpec`` captures both positional and keyword accessible
parameters, but there unfortunately is no object in the runtime that captures
both of these together. Instead, we are forced to separate them into ``*args``
and ``**kwargs``\ , respectively. This means we need to be able to split apart
a single ``ParamSpec`` into these two components, and then bring
them back together into a call. To do this, we introduce ``P.args`` to
represent the tuple of positional arguments in a given call and
``P.kwargs`` to represent the corresponding ``Mapping`` of keywords to
values.
Valid use locations
```````````````````
These "properties" can only be used as the annotated types for
``*args`` and ``**kwargs``\ , accessed from a ParamSpec already in scope.
.. code-block::
def puts_p_into_scope(f: Callable[P, int]) -> None:
def inner(*args: P.args, **kwargs: P.kwargs) -> None: # Accepted
pass
def mixed_up(*args: P.kwargs, **kwargs: P.args) -> None: # Rejected
pass
def misplaced(x: P.args) -> None: # Rejected
pass
def out_of_scope(*args: P.args, **kwargs: P.kwargs) -> None: # Rejected
pass
Furthermore, because the default kind of parameter in Python (\ ``(x: int)``\ )
may be addressed both positionally and through its name, two valid invocations
of a ``(*args: P.args, **kwargs: P.kwargs)`` function may give different
partitions of the same set of parameters. Therefore we need to make sure that
these special types are only brought into the world together, and are used
together, so that our usage is valid for all possible partitions.
.. code-block::
def puts_p_into_scope(f: Callable[P, int]) -> None:
stored_args: P.args # Rejected
stored_kwargs: P.args # Rejected
def just_args(*args: P.args) -> None: # Rejected
pass
def just_kwargs(*args: P.args) -> None: # Rejected
pass
Semantics
`````````
With those requirements met, we can now take advantage of the unique properties With those requirements met, we can now take advantage of the unique properties
afforded to us by this set up: afforded to us by this set up:
* Inside the function, ``args`` has the type ``TParams.args``\ , not * Inside the function, ``args`` has the type ``P.args``\ , not
``Tuple[TParams.args, ...]`` as would be with a normal annotation ``Tuple[P.args, ...]`` as would be with a normal annotation
(and likewise with the ``**kwargs``\ ) (and likewise with the ``**kwargs``\ )
* A function of type ``Callable[TParams, TReturn]`` can be called with * A function of type ``Callable[P, R]`` can be called with ``(*args, **kwargs)``
``(*args, **kwargs)`` if and only if ``args`` has the type ``TParams.args`` if and only if ``args`` has the type ``P.args`` and ``kwargs`` has the type
and ``kwargs`` has the type ``TParams.kwargs``\ , and that those types both ``P.kwargs``\ , and that those types both originated from the same function
originated from the same function declaration. declaration.
* A function declared as * A function declared as ``def inner(*args: P.args, **kwargs: P.kwargs) -> X``
``def inner(*args: TParams.args, **kwargs: TParams.kwargs) -> X`` has type ``Callable[P, X]``.
has type ``Callable[TParams, X]``.
With these three properties, we now have the ability to fully type check With these three properties, we now have the ability to fully type check
parameter preserving decorators. parameter preserving decorators.
One additional form that we want to support is functions that pass only a subset .. code-block::
of their arguments on to another function. To avoid shadowing a named or keyword
only argument in the ``ParameterSpecification`` we require that the additional def decorator(f: Callable[P, int]) -> Callable[P, None]:
arguments be anonymous arguments that precede the ``*args`` and ``*kwargs``
def foo(*args: P.args, **kwargs: P.kwargs) -> None:
f(*args, **kwargs) # Accepted, should resolve to int
f(*kwargs, **args) # Rejected
f(1, *args, **kwargs) # Rejected
return foo # Accepted
To extend this to include ``Concatenate``, we declare the following properties:
* A function of type ``Callable[Concatenate[A, B, P], R]`` can only be
called with ``(a, b, *args, **kwargs)`` when ``args`` and ``kwargs`` are the
respective components of ``P``, ``a`` is of type ``A`` and ``b`` is of
type ``B``.
* A function declared as
``def inner(a: A, b: B, *args: P.args, **kwargs: P.kwargs) -> R``
has type ``Callable[Concatenate[A, B, P], R]``. Placing keyword-only
parameters beterrn the ``*args`` and ``**kwargs`` is forbidden.
.. code-block:: .. code-block::
def call_n_times( def add(f: Callable[P, int]) -> Callable[Concatenate[str, P], None]:
__f: Callable[TParams, None],
__n: int, def foo(s: str, *args: P.args, **kwargs: P.kwargs) -> None: # Accepted
*args: TParams.args, pass
**kwargs: TParams.kwargs,
) -> None: def bar(*args: P.args, s: str, **kwargs: P.kwargs) -> None: # Rejected
for x in range(__n); pass
__f(*args, **kwargs)
return foo # Accepted
def remove(x: Callable[Concatenate[int, P], int]) -> Callable[P, None]:
def foo(*args: P.args, **kwargs: P.kwargs) -> None:
f(1, *args, **kwargs) # Accepted
f(*args, 1, **kwargs) # Rejected
f(*args, **kwargs) # Rejected
return foo
Note that the names of the parameters preceding the ``ParamSpec``
components are not mentioned in the resulting ``Concatenate``. This means that
these parameters can not be addressed via a named argument:
.. code-block::
def outer(f: Callable[P, None]) -> Callable[P, None]:
def foo(x: int, *args: P.args, **kwargs: P.kwargs) -> None:
f(*args, **kwargs)
def bar(*args: P.args, **kwargs: P.kwargs) -> None:
foo(1, *args, **kwargs) # Accepted
foo(x=1, *args, **kwargs) # Rejected
return bar
.. _above:
This is not an implementation convenience, but a soundness requirement. If we
were to allow that second calling style, then the following snippet would be
problematic.
.. code-block::
@outer
def problem(*, x: object) -> None:
pass
problem(x="uh-oh")
Inside of ``bar``, we would get
``TypeError: foo() got multiple values for argument 'x'``. Requiring these
concatenated arguments to be addressed positionally avoids this kind of problem,
and simplifies the syntax for spelling these types. Note that this also why we
have to reject signatures of the form
``(*args: P.args, s: str, **kwargs: P.kwargs)`` (See
`Concatenating Keyword Parameters`_ for more details).
If one of these prepended positional parameters contains a free ``ParamSpec``\ ,
we consider that variable in scope for the purposes of extracting the components
of that ``ParamSpec``. That allows us to spell things like this:
.. code-block::
def twice(f: Callable[P, int], *args: P.args, **kwargs: P.kwargs) -> int:
return f(*args, **kwargs) + f(*args, **kwargs)
The type of ``twice`` in the above example is
``Callable[Concatenate[Callable[P, int], P], int]``, where ``P`` is bound by the
outer ``Callable``. This has the following semantics:
.. code-block::
def a_int_b_str(a: int, b: str) -> int:
pass
twice(a_int_b_str, 1, "A") # Accepted
twice(a_int_b_str, b="A", a=1) # Accepted
twice(a_int_b_str, "A", 1) # Rejected
Backwards Compatibility Backwards Compatibility
----------------------- -----------------------
The only changes necessary to existing features in ``typing`` is allowing these The only changes necessary to existing features in ``typing`` is allowing these
``ParameterSpecification`` objects to be the first parameter to ``Callable`` and ``ParamSpec`` and ``Concatenate`` objects to be the first parameter to
to be a parameter to ``Generic``. Currently ``Callable`` expects a list of types ``Callable`` and to be a parameter to ``Generic``. Currently ``Callable``
there and ``Generic`` expects single types, so they are currently mutually expects a list of types there and ``Generic`` expects single types, so they are
exclusive. Otherwise, existing code that doesn't reference the new interfaces currently mutually exclusive. Otherwise, existing code that doesn't reference
will be unaffected. the new interfaces will be unaffected.
Reference Implementation Reference Implementation
------------------------ ------------------------
The `Pyre <https://pyre-check.org/>`_ type checker supports The `Pyre <https://pyre-check.org/>`_ type checker supports all of the behavior
``ParameterSpecification``\ s, ``.args`` and ``.kwargs`` in the context of described above. A reference implementation of the runtime components needed
functions. Support for use with ``Generic`` is not yet implemented. A reference for those uses is provided in the ``pyre_extensions`` module.
implementation of the runtime components needed for those uses is provided in
the ``pyre_extensions`` module.
Rejected Alternatives Rejected Alternatives
--------------------- ---------------------
@ -304,11 +557,11 @@ so:
.. code-block:: .. code-block::
Treturn = typing.TypeVar(“Treturn”) R = typing.TypeVar(“R”)
Tpositionals = .... Tpositionals = ....
Tkeywords = ... Tkeywords = ...
class BetterCallable(typing.Protocol[Tpositionals, Tkeywords, Treturn]): class BetterCallable(typing.Protocol[Tpositionals, Tkeywords, R]):
def __call__(*args: Tpositionals, **kwargs: Tkeywords) -> Treturn: ... def __call__(*args: Tpositionals, **kwargs: Tkeywords) -> R: ...
However there are some problems with trying to come up with a consistent However there are some problems with trying to come up with a consistent
solution for those type variables for a given callable. This problem comes up solution for those type variables for a given callable. This problem comes up
@ -321,9 +574,9 @@ with even the simplest of callables:
simple <: BetterCallable[[], {“x”: int}, None] simple <: BetterCallable[[], {“x”: int}, None]
BetterCallable[[int], [], None] </: BetterCallable[[], {“x”: int}, None] BetterCallable[[int], [], None] </: BetterCallable[[], {“x”: int}, None]
Any time where a type can implement a protocol in more than one way that aren’t Any time where a type can implement a protocol in more than one way that aren't
mutually compatible, we can run into situations where we lose information. If we mutually compatible, we can run into situations where we lose information. If we
were to make a decorator using this protocol, we have to pick one calling were to make a decorator using this protocol, we would have to pick one calling
convention to prefer. convention to prefer.
.. code-block:: .. code-block::
@ -332,43 +585,162 @@ convention to prefer.
f: BetterCallable[[Ts], [Tmap], int], f: BetterCallable[[Ts], [Tmap], int],
) -> BetterCallable[[Ts], [Tmap], str]: ) -> BetterCallable[[Ts], [Tmap], str]:
def decorated(*args: Ts, **kwargs: Tmap) -> str: def decorated(*args: Ts, **kwargs: Tmap) -> str:
x = f(*args, **kwargs) x = f(*args, **kwargs)
return int_to_str(x) return int_to_str(x)
return decorated return decorated
@decorator @decorator
def foo(x: int) -> int: def foo(x: int) -> int:
return x return x
reveal_type(foo) # Option A: BetterCallable[[int], {}, str] reveal_type(foo) # Option A: BetterCallable[[int], {}, str]
# Option B: BetterCallable[[], {x: int}, str] # Option B: BetterCallable[[], {x: int}, str]
foo(7) # fails under option B foo(7) # fails under option B
foo(x=7) # fails under option A foo(x=7) # fails under option A
The core problem here is that, by default, parameters in Python can either be The core problem here is that, by default, parameters in Python can either be
passed in positionally or as a keyword parameter. This means we really have called positionally or as a keyword argument. This means we really have
three categories (positional-only, positional-or-keyword, keyword-only) we’re three categories (positional-only, positional-or-keyword, keyword-only) we’re
trying to jam into two categories. This is the same problem that we briefly trying to jam into two categories. This is the same problem that we briefly
mentioned when discussing ``.args`` and ``.kwargs``. Fundamentally, in order to mentioned when discussing ``.args`` and ``.kwargs``. Fundamentally, in order to
capture two categories when there are some things that can be in either capture two categories when there are some things that can be in either
category, we need a higher level primitive (\ ``ParameterSpecification``\ ) to category, we need a higher level primitive (\ ``ParamSpec``\ ) to
capture all three, and then split them out afterward. capture all three, and then split them out afterward.
Mutations on ParameterSpecifications Defining ParametersOf
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^
There are still a class of decorators still not supported with these features: Another proposal we considered was defining ``ParametersOf`` and ``ReturnType``
those that mutate (add/remove/change) the parameters of the given function. operators which would operate on a domain of a newly defined ``Function`` type.
Defining operators that do these mutations becomes very complicated very ``Function`` would be callable with, and only with ``ParametersOf[F]``.
quickly, as you have to deal with name collision issues much more prominently. ``ParametersOf`` and ``ReturnType`` would only operate on type variables with
We will defer that work until there is significant demand, and then we would be precisely this bound. The combination of these three features could express
open to revisiting it. everything that we can express with ``ParamSpecs``.
.. code-block::
F = TypeVar("F", bound=Function)
def no_change(f: F) -> F:
def inner(
*args: ParametersOf[F].args,
**kwargs: ParametersOf[F].kwargs
) -> ReturnType[F]:
return f(*args, **kwargs)
return inner
def wrapping(f: F) -> Callable[ParametersOf[F], List[ReturnType[F]]]:
def inner(
*args: ParametersOf[F].args,
**kwargs: ParametersOf[F].kwargs
) -> List[ReturnType[F]]:
return [f(*args, **kwargs)]
return inner
def unwrapping(
f: Callable[ParametersOf[F], List[R]]
) -> Callable[ParametersOf[F], R]:
def inner(
*args: ParametersOf[F].args,
**kwargs: ParametersOf[F].kwargs
) -> R:
return f(*args, **kwargs)[0]
return inner
We decided to go with ``ParamSpec``\ s over this approach for several reasons:
* The footprint of this change would be larger, as we would need two new
operators, and a new type, while ``ParamSpec`` just introduces a new variable.
* Python typing has so far has avoided supporting operators, whether
user-defined or built-in, in favor of destructuring. Accordingly,
``ParamSpec`` based signatures look much more like existing Python.
* The lack of user-defined operators makes common patterns hard to spell.
``unwrapping`` is odd to read because ``F`` is not actually referring to any
callable. It’s just being used as a container for the parameters we wish to
propagate. It would read better if we could define an operator
``RemoveList[List[X]] = X`` and then ``unwrapping`` could take ``F`` and
return ``Callable[ParametersOf[F], RemoveList[ReturnType[F]]]``. Without
that, we unfortunately get into a situation where we have to use a
``Function``-variable as an improvised ``ParamSpec``, in that we never
actually bind the return type.
In summary, between these two equivalently powerful syntaxes, ``ParamSpec`` fits
much more naturally into the status quo.
.. _Concatenating Keyword Parameters:
Concatenating Keyword Parameters
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In principle the idea of concatenation as a means to modify a finite number of
positional parameters could be expanded to include keyword parameters.
.. code-block::
def add_n(f: Callable[P, R]) -> Callable[Concatenate[("n", int), P], R]:
def inner(*args: P.args, n: int, **kwargs: P.kwargs) -> R:
# use n
return f(*args, **kwargs)
return inner
However, the key distinction is that while prepending positional-only parameters
to a valid callable type always yields another valid callable type, the same
cannot be said for adding keyword-only parameters. As alluded to above_ , the
issue is name collisions. The parameters ``Concatenate[("n", int), P]`` are
only valid when ``P`` itself does not already have a parameter named ``n``\ .
.. code-block::
def innocent_wrapper(f: Callable[P, R]) -> Callable[P, R]:
def inner(*args: P.args, **kwargs: P.kwargs) -> R:
added = add_n(f)
return added(*args, n=1, **kwargs)
return inner
@innocent_wrapper
def problem(n: int) -> None:
pass
Calling ``problem(2)`` works fine, but calling ``problem(n=2)`` leads to a
``TypeError: problem() got multiple values for argument 'n'`` from the call to
``added`` inside of ``innocent_wrapper``\ .
This kind of situation could be avoided, and this kind of decorator could be
typed if we could reify the constraint that a set of parameters **not** contain
a certain name, with something like:
.. code-block::
P_without_n = ParamSpec("P_without_n", banned_names=["n"])
def add_n(
f: Callable[P_without_n, R]
) -> Callable[Concatenate[("n", int), P_without_n], R]: ...
The call to ``add_n`` inside of ``innocent_wrapper`` could then be rejected
since the callable was not guaranteed not to already have a parameter named
``n``\ .
However, enforcing these constraints would require enough additional
implementation work that we judged this extension to be out of scope of this
PEP. Fortunately the design of ``ParamSpec``\ s are such that we can return to
this idea later if there is sufficient demand.
Naming this a ``ParameterSpecification``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
We decided that ParameterSpecification was a little too long-winded for use
here, and that this style of abbreviated name made it look more like TypeVar.
Naming this an ``ArgSpec`` Naming this an ``ArgSpec``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^
We think that calling this a ParameterSpecification is more correct than We think that calling this a ParamSpec is more correct than
referring to it as an Argument Specification, since callables have parameters, referring to it as an ArgSpec, since callables have parameters,
which are distinct from the arguments which are passed to them in a given call which are distinct from the arguments which are passed to them in a given call
site. A given binding for a ParameterSpecification is a set of function site. A given binding for a ParamSpec is a set of function
parameters, not a call-site’s arguments. parameters, not a call-site’s arguments.
Acknowledgements Acknowledgements
@ -384,5 +756,5 @@ more compact ``.args``\ /\ ``.kwargs`` syntax.
Copyright Copyright
--------- ---------
This document is placed in the public domain or under the CC0-1.0-Universal This document is placed in the public domain or under the CC0-1.0-Universal
license, whichever is more permissive. license, whichever is more permissive.