389 lines
15 KiB
ReStructuredText
389 lines
15 KiB
ReStructuredText
PEP: 612
|
||
Title: Parameter Specification Variables
|
||
Author: Mark Mendoza <mendoza.mark.a@gmail.com>
|
||
Sponsor: Guido van Rossum <guido@python.org>
|
||
Discussions-To: Typing-Sig <typing-sig@python.org>
|
||
Status: Draft
|
||
Type: Standards Track
|
||
Content-Type: text/x-rst
|
||
Created: 18-Dec-2019
|
||
Python-Version: 3.9
|
||
Post-History: 18-Dec-2019
|
||
|
||
Parameter Specification Variables
|
||
=================================
|
||
|
||
Abstract
|
||
--------
|
||
|
||
There currently are two ways to specify the type of a callable, the
|
||
``Callable[[T1, T2], TReturn]`` syntax defined in `PEP 484
|
||
<https://www.python.org/dev/peps/pep-0484>`_\ , and callback protocols from `PEP
|
||
544 <https://www.python.org/dev/peps/pep-0544/#callback-protocols>`_. Neither of
|
||
these support forwarding the parameter types of one callable over to another
|
||
callable, making it difficult to annotate function decorators. This PEP proposes
|
||
``typing.ParameterSpecification``\ , a new kind of type variable, to support
|
||
expressing these kinds of relationships.
|
||
|
||
Motivation
|
||
----------
|
||
|
||
The existing standards for annotating higher order functions don’t give us the
|
||
tools to annotate the following common decorator pattern satisfactorily:
|
||
|
||
.. code-block::
|
||
|
||
from typing import Awaitable, Callable, TypeVar
|
||
|
||
TReturn = TypeVar("TReturn")
|
||
|
||
def add_logging(
|
||
f: Callable[..., TReturn]
|
||
) -> Callable[..., Awaitable[TReturn]]:
|
||
async def inner(*args: object, **kwargs: object) -> TReturn:
|
||
await log_to_database()
|
||
return f(*args, **kwargs)
|
||
return inner
|
||
|
||
@add_logging
|
||
def foo(x: int, y: str) -> int:
|
||
return x + 7
|
||
|
||
await foo(1, "A")
|
||
await foo("B", 2) # fails at runtime
|
||
|
||
``add_logging``\ , a decorator which logs before each entry into the decorated
|
||
function, is an instance of the Python idiom of one function passing all
|
||
arguments given to it over to another function through the combination of the
|
||
``*args`` and ``**kwargs`` features in both parameters and in arguments. When
|
||
one defines a function (like ``inner``\ ) that takes ``(*args, **kwargs)`` and
|
||
goes on to call another function with ``(*args, **kwargs)``\ , the wrapping
|
||
function can only be safely called in all of the ways that the wrapped function
|
||
could be safely called. To type this decorator, we’d like to be able to place
|
||
a dependency between the parameters of the callable ``f`` and the parameters of
|
||
the returned function. `PEP 484 <https://www.python.org/dev/peps/pep-0484>`_
|
||
supports dependencies between single types, as in ``def append(l:
|
||
typing.List[T], e: T) -> typing.List[T]: ...``\ , but there is no existing way
|
||
to do so with a complicated entity like the parameters one could pass to
|
||
a function.
|
||
|
||
Due to the limitations of the status quo, the ``add_logging`` example will type
|
||
check but will fail at runtime. ``inner`` will pass the string “B” into ``foo``\
|
||
, which will try to add 7 to it, triggering a type error. This was not caught
|
||
by the type checker because the decorated ``foo`` was given the type
|
||
``Callable[..., Awaitable[int]]`` which is specified to do no validation on its
|
||
arguments.
|
||
|
||
Without the ability to define dependencies between the parameters of different
|
||
callable types, there is no way, at present, to make ``add_logging`` compatible
|
||
with all functions, while still preserving the enforcement of the parameters of
|
||
the decorated function.
|
||
|
||
With the addition of the ``ParameterSpecification`` variables proposed by this
|
||
PEP, we can rewrite the previous example in a way that keeps the flexibility of
|
||
the decorator and the parameter enforcement of the decorated function.
|
||
|
||
.. code-block::
|
||
|
||
from typing import Awaitable, Callable, ParameterSpecification, TypeVar
|
||
|
||
Ps = ParameterSpecification("Ps")
|
||
R = TypeVar("R")
|
||
|
||
def add_logging(f: Callable[Ps, R]) -> Callable[Ps, Awaitable[R]]:
|
||
async def inner(*args: Ps.args, **kwargs: Ps.kwargs) -> R:
|
||
await log_to_database()
|
||
return f(*args, **kwargs)
|
||
|
||
return inner
|
||
|
||
@add_logging
|
||
def foo(x: int, y: str) -> int:
|
||
return x + 7
|
||
|
||
await foo(1, "A")
|
||
await foo("B", 2) # Incompatible parameter type:
|
||
# Expected `int` for 1st anonymous parameter to call `foo`
|
||
# but got `str`
|
||
|
||
Specification
|
||
-------------
|
||
|
||
Declarations
|
||
^^^^^^^^^^^^
|
||
|
||
A parameter specification variable is defined in a similar manner to a normal
|
||
``typing.TypeVar``.
|
||
|
||
.. code-block::
|
||
|
||
from typing import ParameterSpecification
|
||
TParams = ParameterSpecification("TParams") # Accepted
|
||
TParams = ParameterSpecification("WrongName") # Rejected
|
||
|
||
The runtime should accept ``bound``\ s and ``covariant`` and ``contravariant``
|
||
arguments in the declaration just as ``typing.TypeVar`` does, but for now we
|
||
will defer the standardization of the semantics of those options to a later PEP.
|
||
|
||
Valid use locations
|
||
^^^^^^^^^^^^^^^^^^^
|
||
|
||
A declared ``ParameterSpecification`` can only be used in the place of the list
|
||
of types in the declaration of a ``Callable`` type, or a user defined class
|
||
which is generic in a ``ParameterSpecification`` variable (i.e., ``MyClass`` in
|
||
the following example).
|
||
|
||
.. code-block::
|
||
|
||
def foo(
|
||
x: typing.Callable[TParams, int]
|
||
) -> typing.Callable[TParams, str]: # Accepted
|
||
...
|
||
def foo(
|
||
x: MyClass[TParams, int]
|
||
) -> typing.Callable[TParams, str]: # Accepted
|
||
...
|
||
def foo(x: TParams) -> TParams: ... # Rejected
|
||
def foo(x: typing.List[TParams]) -> None: ... # Rejected
|
||
def foo(x: typing.Callable[[int, str], TParams]) -> None: ... # Rejected
|
||
|
||
Semantics
|
||
^^^^^^^^^
|
||
|
||
The inference rules for the return type of a function invocation whose signature
|
||
contains a ``ParameterSpecification`` variable are analogous to those around
|
||
evaluating ones with ``TypeVar``\ s.
|
||
|
||
.. code-block::
|
||
|
||
def foo(
|
||
x: typing.Callable[TParams, int]
|
||
) -> typing.Callable[TParams, str]: ...
|
||
def bar(a: str, b: bool) -> int: ...
|
||
f = foo(bar) # f should be inferred to have the same signature as bar,
|
||
# but returning str
|
||
f("A", True) # Accepted
|
||
f(a="A", b=True) # Accepted
|
||
f("A", "A") # Rejected
|
||
|
||
Just as with traditional ``TypeVars``\ , a user may include the same
|
||
``ParameterSpecification`` multiple times in the arguments of the same function,
|
||
to indicate a dependency between multiple arguments. In these cases a type
|
||
checker may choose to solve to a common behavioral supertype (i.e. a set of
|
||
parameters for which all of the valid calls are valid in both of the subtypes),
|
||
but is not obligated to do so.
|
||
|
||
.. code-block::
|
||
|
||
def foo(
|
||
x: typing.Callable[TParams, int], y: typing.Callable[TParams, int]
|
||
) -> typing.Callable[TParams, bool]: ...
|
||
|
||
def x_int_y_str(x: int, y: str) -> int: ...
|
||
def y_int_x_str(y: int, x: str) -> int: ...
|
||
foo(x_int_y_str, x_int_y_str) # Must return (x: int, y: str) -> int
|
||
foo(x_int_y_str, y_int_x_str) # Could return (__a: int, __b: str) -> int
|
||
# This works because both callables have types
|
||
# that are behavioral subtypes of
|
||
# Callable[[int, str], int]
|
||
|
||
def keyword_only_x(*, x: int) -> int: ...
|
||
def keyword_only_y(*, y: int) -> int: ...
|
||
foo(keyword_only_x, keyword_only_y) # Must be rejected
|
||
|
||
Use in ``Generic`` Classes
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
Just as with normal ``TypeVar``\ s, ``ParameterSpecification``\ s can be used to
|
||
make generic classes as well as generic functions. These are able to be
|
||
mixed with normal ``TypeVar``\ s. This also work with
|
||
protocols in the same manner.
|
||
|
||
The components of a ``ParameterSpecification``
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
A ``ParameterSpecification`` captures both positional and keyword accessible
|
||
parameters, but there unfortunately is no object in the runtime that captures
|
||
both of these together. Instead, we are forced to separate them into ``*args``
|
||
and ``**kwargs``\ , respectively. This means we need to be able to split apart
|
||
a single ``ParameterSpecification`` into these two components, and then bring
|
||
them back together into a call. To do this, we introduce ``TParams.args`` to
|
||
represent the tuple of positional arguments in a given call and
|
||
``TParams.kwargs`` to represent the corresponding ``Mapping`` of keywords to
|
||
values. These operators can only be used together, as the annotated types for
|
||
``*args`` and ``**kwargs`` .
|
||
|
||
.. code-block::
|
||
|
||
class G(Generic[TParams]):
|
||
def foo(
|
||
*args: TParams.args, **kwargs: TParams.kwargs
|
||
) -> int: # Accepted
|
||
...
|
||
def bar(
|
||
*args: TParams.kwargs, **kwargs: TParams.args
|
||
) -> int: # Rejected
|
||
...
|
||
def baz(*args: TParams.args) -> int: ... # Rejected
|
||
stored_arguments: TParams.args # Rejected
|
||
def bap(x: TParams.args) -> int: ... # Rejected
|
||
def bop(
|
||
*args: List[TParams.args], **kwargs: TParams.kwargs
|
||
) -> int: # Rejected
|
||
...
|
||
|
||
Because the default kind of parameter in Python (\ ``(x: int)``\ ) may be
|
||
addressed both positionally and through its name, two valid invocations of
|
||
a ``(*args: TParams.args, **kwargs: TParams.kwargs)`` function may give
|
||
different partitions of the same set of parameters. Therefore we need to make
|
||
sure that these special types are only brought into the world together, and are
|
||
used together, so that our usage is valid for all possible partitions.
|
||
|
||
With those requirements met, we can now take advantage of the unique properties
|
||
afforded to us by this set up:
|
||
|
||
|
||
* Inside the function, ``args`` has the type ``TParams.args``\ , not
|
||
``Tuple[TParams.args, ...]`` as would be with a normal annotation
|
||
(and likewise with the ``**kwargs``\ )
|
||
* A function of type ``Callable[TParams, TReturn]`` can be called with
|
||
``(*args, **kwargs)`` if and only if ``args`` has the type ``TParams.args``
|
||
and ``kwargs`` has the type ``TParams.kwargs``\ , and that those types both
|
||
originated from the same function declaration.
|
||
* A function declared as
|
||
``def inner(*args: TParams.args, **kwargs: TParams.kwargs) -> X``
|
||
has type ``Callable[TParams, X]``.
|
||
|
||
With these three properties, we now have the ability to fully type check
|
||
parameter preserving decorators.
|
||
|
||
One additional form that we want to support is functions that pass only a subset
|
||
of their arguments on to another function. To avoid shadowing a named or keyword
|
||
only argument in the ``ParameterSpecification`` we require that the additional
|
||
arguments be anonymous arguments that precede the ``*args`` and ``*kwargs``
|
||
|
||
.. code-block::
|
||
|
||
def call_n_times(
|
||
__f: Callable[TParams, None],
|
||
__n: int,
|
||
*args: TParams.args,
|
||
**kwargs: TParams.kwargs,
|
||
) -> None:
|
||
for x in range(__n);
|
||
__f(*args, **kwargs)
|
||
|
||
Backwards Compatibility
|
||
-----------------------
|
||
|
||
The only changes necessary to existing features in ``typing`` is allowing these
|
||
``ParameterSpecification`` objects to be the first parameter to ``Callable`` and
|
||
to be a parameter to ``Generic``. Currently ``Callable`` expects a list of types
|
||
there and ``Generic`` expects single types, so they are currently mutually
|
||
exclusive. Otherwise, existing code that doesn't reference the new interfaces
|
||
will be unaffected.
|
||
|
||
Reference Implementation
|
||
------------------------
|
||
|
||
The `Pyre <https://pyre-check.org/>`_ type checker supports
|
||
``ParameterSpecification``\ s, ``.args`` and ``.kwargs`` in the context of
|
||
functions. Support for use with ``Generic`` is not yet implemented. A reference
|
||
implementation of the runtime components needed for those uses is provided in
|
||
the ``pyre_extensions`` module.
|
||
|
||
Rejected Alternatives
|
||
---------------------
|
||
|
||
Using List Variadics and Map Variadics
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
We considered just trying to make something like this with a callback protocol
|
||
which was parameterized on a list-type variadic, and a map-type variadic like
|
||
so:
|
||
|
||
.. code-block::
|
||
|
||
Treturn = typing.TypeVar(“Treturn”)
|
||
Tpositionals = ....
|
||
Tkeywords = ...
|
||
class BetterCallable(typing.Protocol[Tpositionals, Tkeywords, Treturn]):
|
||
def __call__(*args: Tpositionals, **kwargs: Tkeywords) -> Treturn: ...
|
||
|
||
However there are some problems with trying to come up with a consistent
|
||
solution for those type variables for a given callable. This problem comes up
|
||
with even the simplest of callables:
|
||
|
||
.. code-block::
|
||
|
||
def simple(x: int) -> None: ...
|
||
simple <: BetterCallable[[int], [], None]
|
||
simple <: BetterCallable[[], {“x”: int}, None]
|
||
BetterCallable[[int], [], None] </: BetterCallable[[], {“x”: int}, None]
|
||
|
||
Any time where a type can implement a protocol in more than one way that aren’t
|
||
mutually compatible, we can run into situations where we lose information. If we
|
||
were to make a decorator using this protocol, we have to pick one calling
|
||
convention to prefer.
|
||
|
||
.. code-block::
|
||
|
||
def decorator(
|
||
f: BetterCallable[[Ts], [Tmap], int],
|
||
) -> BetterCallable[[Ts], [Tmap], str]:
|
||
def decorated(*args: Ts, **kwargs: Tmap) -> str:
|
||
x = f(*args, **kwargs)
|
||
return int_to_str(x)
|
||
return decorated
|
||
@decorator
|
||
def foo(x: int) -> int:
|
||
return x
|
||
reveal_type(foo) # Option A: BetterCallable[[int], {}, str]
|
||
# Option B: BetterCallable[[], {x: int}, str]
|
||
foo(7) # fails under option B
|
||
foo(x=7) # fails under option A
|
||
|
||
The core problem here is that, by default, parameters in Python can either be
|
||
passed in positionally or as a keyword parameter. This means we really have
|
||
three categories (positional-only, positional-or-keyword, keyword-only) we’re
|
||
trying to jam into two categories. This is the same problem that we briefly
|
||
mentioned when discussing ``.args`` and ``.kwargs``. Fundamentally, in order to
|
||
capture two categories when there are some things that can be in either
|
||
category, we need a higher level primitive (\ ``ParameterSpecification``\ ) to
|
||
capture all three, and then split them out afterward.
|
||
|
||
Mutations on ParameterSpecifications
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
There are still a class of decorators still not supported with these features:
|
||
those that mutate (add/remove/change) the parameters of the given function.
|
||
Defining operators that do these mutations becomes very complicated very
|
||
quickly, as you have to deal with name collision issues much more prominently.
|
||
We will defer that work until there is significant demand, and then we would be
|
||
open to revisiting it.
|
||
|
||
Naming this an ``ArgSpec``
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
We think that calling this a ParameterSpecification is more correct than
|
||
referring to it as an Argument Specification, since callables have parameters,
|
||
which are distinct from the arguments which are passed to them in a given call
|
||
site. A given binding for a ParameterSpecification is a set of function
|
||
parameters, not a call-site’s arguments.
|
||
|
||
Acknowledgements
|
||
----------------
|
||
|
||
Thanks to all of the members of the Pyre team for their comments on early drafts
|
||
of this PEP, and for their help with the reference implementation.
|
||
|
||
Thanks are also due to the whole Python typing community for their early
|
||
feedback on this idea at a Python typing meetup, leading directly to the much
|
||
more compact ``.args``\ /\ ``.kwargs`` syntax.
|
||
|
||
Copyright
|
||
---------
|
||
|
||
This document is placed in the public domain or under the CC0-1.0-Universal
|
||
license, whichever is more permissive.
|