iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

PEP 612: Formatting changes (#1260) 

* line breaking

* one extra space

* Ps

* fix syntax
This commit is contained in:
Mark Mendoza 2019-12-20 13:12:00 -08:00 committed by Chris Angelico
parent 0644091b0e
commit 50eadddca7
1 changed files with 176 additions and 47 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

223
pep-0612.rst
View File

@ -15,12 +15,20 @@ Parameter Specification Variables
Abstract
--------
There currently are two ways to specify the type of a callable, the ``Callable[[T1, T2], TReturn]`` syntax defined in `PEP 484 <https://www.python.org/dev/peps/pep-0484>`_\ , and callback protocols from `PEP 544 <https://www.python.org/dev/peps/pep-0544/#callback-protocols>`_. Neither of these support forwarding the parameter types of one callable over to another callable, making it difficult to annotate function decorators. This PEP proposes ``typing.ParameterSpecification``\ , a new kind of type variable, to support expressing these kinds of relationships.
There currently are two ways to specify the type of a callable, the
``Callable[[T1, T2], TReturn]`` syntax defined in `PEP 484
<https://www.python.org/dev/peps/pep-0484>`_\ , and callback protocols from `PEP
544 <https://www.python.org/dev/peps/pep-0544/#callback-protocols>`_. Neither of
these support forwarding the parameter types of one callable over to another
callable, making it difficult to annotate function decorators. This PEP proposes
``typing.ParameterSpecification``\ , a new kind of type variable, to support
expressing these kinds of relationships.
Motivation
----------
The existing standards for annotating higher order functions don’t give us the tools to annotate the following common decorator pattern satisfactorily:
The existing standards for annotating higher order functions don’t give us the
tools to annotate the following common decorator pattern satisfactorily:
.. code-block::
@ -28,7 +36,9 @@ The existing standards for annotating higher order functions don’t give us the
TReturn = TypeVar("TReturn")
def add_logging(f: Callable[..., TReturn]) -> Callable[..., Awaitable[TReturn]]:
def add_logging(
f: Callable[..., TReturn]
) -> Callable[..., Awaitable[TReturn]]:
async def inner(*args: object, **kwargs: object) -> TReturn:
await log_to_database()
return f(*args, **kwargs)
@ -41,19 +51,42 @@ The existing standards for annotating higher order functions don’t give us the
await foo(1, "A")
await foo("B", 2) # fails at runtime
``add_logging``\ , a decorator which logs before each entry into the decorated function, is an instance of the Python idiom of one function passing all arguments given to it over to another function through the combination of the ``*args`` and ``**kwargs`` features in both parameters and in arguments. When one defines a function (like ``inner``\ ) that takes ``(*args, **kwargs)`` and goes on to call another function with ``(*args, **kwargs)``\ , the wrapping function can only be safely called in all of the ways that the wrapped function could be safely called. To type this decorator, we’d like to be able to place a dependency between the parameters of the callable ``f`` and the parameters of the returned function. `PEP 484 <https://www.python.org/dev/peps/pep-0484>`_ supports dependencies between single types, as in ``def append(l: typing.List[T], e: T) -> typing.List[T]: ...``\ , but there is no existing way to do so with a complicated entity like the parameters one could pass to a function.
``add_logging``\ , a decorator which logs before each entry into the decorated
function, is an instance of the Python idiom of one function passing all
arguments given to it over to another function through the combination of the
``*args`` and ``**kwargs`` features in both parameters and in arguments. When
one defines a function (like ``inner``\ ) that takes ``(*args, **kwargs)`` and
goes on to call another function with ``(*args, **kwargs)``\ , the wrapping
function can only be safely called in all of the ways that the wrapped function
could be safely called. To type this decorator, we’d like to be able to place
a dependency between the parameters of the callable ``f`` and the parameters of
the returned function. `PEP 484 <https://www.python.org/dev/peps/pep-0484>`_
supports dependencies between single types, as in ``def append(l:
typing.List[T], e: T) -> typing.List[T]: ...``\ , but there is no existing way
to do so with a complicated entity like the parameters one could pass to
a function.
Due to the limitations of the status quo, the ``add_logging`` example will type check but will fail at runtime. ``inner`` will pass the string “B” into ``foo``\ , which will try to add 7 to it, triggering a type error. This was not caught by the type checker because the decorated ``foo`` was given the type ``Callable[..., Awaitable[int]]`` which is specified to do no validation on its arguments.
Due to the limitations of the status quo, the ``add_logging`` example will type
check but will fail at runtime. ``inner`` will pass the string “B” into ``foo``\
, which will try to add 7 to it, triggering a type error. This was not caught
by the type checker because the decorated ``foo`` was given the type
``Callable[..., Awaitable[int]]`` which is specified to do no validation on its
arguments.
Without the ability to define dependencies between the parameters of different callable types, there is no way, at present, to make ``add_logging`` compatible with all functions, while still preserving the enforcement of the parameters of the decorated function.
Without the ability to define dependencies between the parameters of different
callable types, there is no way, at present, to make ``add_logging`` compatible
with all functions, while still preserving the enforcement of the parameters of
the decorated function.
With the addition of the ``ParameterSpecification`` variables proposed by this PEP, we can rewrite the previous example in a way that keeps the flexibility of the decorator and the parameter enforcement of the decorated function.
With the addition of the ``ParameterSpecification`` variables proposed by this
PEP, we can rewrite the previous example in a way that keeps the flexibility of
the decorator and the parameter enforcement of the decorated function.
.. code-block::
from typing import Awaitable, Callable, ParameterSpecification, TypeVar
Ps = ParameterSpecification("TParams")
Ps = ParameterSpecification("Ps")
R = TypeVar("TReturn")
def add_logging(f: Callable[Ps, R]) -> Callable[Ps, Awaitable[R]]:
@ -68,8 +101,9 @@ With the addition of the ``ParameterSpecification`` variables proposed by this P
return x + 7
await foo(1, "A")
await foo("B", 2) # Incompatible parameter type: Expected `int` for 1st anonymous parameter
# to call `foo` but got `str`
await foo("B", 2) # Incompatible parameter type:
# Expected `int` for 1st anonymous parameter to call `foo`
# but got `str`
Specification
-------------
@ -77,7 +111,8 @@ Specification
Declarations
^^^^^^^^^^^^
A parameter specification variable is defined in a similar manner to a normal ``typing.TypeVar``.
A parameter specification variable is defined in a similar manner to a normal
``typing.TypeVar``.
.. code-block::
@ -85,36 +120,57 @@ A parameter specification variable is defined in a similar manner to a normal ``
TParams = ParameterSpecification("TParams") # Accepted
TParams = ParameterSpecification("WrongName") # Rejected
The runtime should accept ``bound``\ s and ``covariant`` and ``contravariant`` arguments in the declaration just as ``typing.TypeVar`` does, but for now we will defer the standardization of the semantics of those options to a later PEP.
The runtime should accept ``bound``\ s and ``covariant`` and ``contravariant``
arguments in the declaration just as ``typing.TypeVar`` does, but for now we
will defer the standardization of the semantics of those options to a later PEP.
Valid use locations
^^^^^^^^^^^^^^^^^^^
A declared ``ParameterSpecification`` can only be used in the place of the list of types in the declaration of a ``Callable`` type, or a user defined class which is generic in a ``ParameterSpecification`` variable (i.e., ``MyClass`` in the following example).
A declared ``ParameterSpecification`` can only be used in the place of the list
of types in the declaration of a ``Callable`` type, or a user defined class
which is generic in a ``ParameterSpecification`` variable (i.e., ``MyClass`` in
the following example).
.. code-block::
def foo(x: typing.Callable[TParams, int]) -> typing.Callable[TParams, str]: ... # Accepted
def foo(x: MyClass[TParams, int]) -> typing.Callable[TParams, str]: ... # Accepted
def foo(x: TParams) -> TParams: ... # Rejected
def foo(
x: typing.Callable[TParams, int]
) -> typing.Callable[TParams, str]: # Accepted
...
def foo(
x: MyClass[TParams, int]
) -> typing.Callable[TParams, str]: # Accepted
...
def foo(x: TParams) -> TParams: ... # Rejected
def foo(x: typing.List[TParams]) -> None: ... # Rejected
def foo(x: typing.Callable[[int, str], TParams]) -> None: ... # Rejected
Semantics
^^^^^^^^^
The inference rules for the return type of a function invocation whose signature contains a ``ParameterSpecification`` variable are analogous to those around evaluating ones with ``TypeVar``\ s.
The inference rules for the return type of a function invocation whose signature
contains a ``ParameterSpecification`` variable are analogous to those around
evaluating ones with ``TypeVar``\ s.
.. code-block::
def foo(x: typing.Callable[TParams, int]) -> typing.Callable[TParams, str]: ...
def foo(
x: typing.Callable[TParams, int]
) -> typing.Callable[TParams, str]: ...
def bar(a: str, b: bool) -> int: ...
f = foo(bar) # f should be inferred to have the same signature as bar, but returning str
f = foo(bar) # f should be inferred to have the same signature as bar,
# but returning str
f("A", True) # Accepted
f(a = "A", b = True) # Accepted
f("A", "A") # Rejected
Just as with traditional ``TypeVars``\ , a user may include the same ``ParameterSpecification`` multiple times in the arguments of the same function, to indicate a dependency between multiple arguments. In these cases a type checker may choose to solve to a common behavioral supertype (i.e. a set of parameters for which all of the valid calls are valid in both of the subtypes), but is not obligated to do so.
Just as with traditional ``TypeVars``\ , a user may include the same
``ParameterSpecification`` multiple times in the arguments of the same function,
to indicate a dependency between multiple arguments. In these cases a type
checker may choose to solve to a common behavioral supertype (i.e. a set of
parameters for which all of the valid calls are valid in both of the subtypes),
but is not obligated to do so.
.. code-block::
@ -126,8 +182,9 @@ Just as with traditional ``TypeVars``\ , a user may include the same ``Parameter
def y_int_x_str(y: int, x: str) -> int: ...
foo(x_int_y_str, x_int_y_str) # Must return (x: int, y: str) -> int
foo(x_int_y_str, y_int_x_str) # Could return (__a: int, __b: str) -> int
# This works because both callables have types that are behavioral
# subtypes of Callable[[int, str], int]
# This works because both callables have types
# that are behavioral subtypes of
# Callable[[int, str], int]
def keyword_only_x(*, x: int) -> int: ...
def keyword_only_y(*, y: int) -> int: ...
@ -136,37 +193,73 @@ Just as with traditional ``TypeVars``\ , a user may include the same ``Parameter
Use in ``Generic`` Classes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Just as with normal ``TypeVar``\ s, ``ParameterSpecification``\ s can be used to make generic classes as well as generic functions. These ought to be able to be mixed with normal ``TypeVar``\ s. This should also be made to work with protocols in the same manner.
Just as with normal ``TypeVar``\ s, ``ParameterSpecification``\ s can be used to
make generic classes as well as generic functions. These ought to be able to be
mixed with normal ``TypeVar``\ s. This should also be made to work with
protocols in the same manner.
The components of a ``ParameterSpecification``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A ``ParameterSpecification`` captures both positional and keyword accessible parameters, but there unfortunately is no object in the runtime that captures both of these together. Instead, we are forced to separate them into ``*args`` and ``**kwargs``\ , respectively. This means we need to be able to split apart a single ``ParameterSpecification`` into these two components, and then bring them back together into a call. To do this, we introduce ``TParams.args`` to represent the tuple of positional arguments in a given call and ``TParams.kwargs`` to represent the corresponding ``Mapping`` of keywords to values. These operators can only be used together, as the annotated types for ``*args`` and ``**kwargs`` .
A ``ParameterSpecification`` captures both positional and keyword accessible
parameters, but there unfortunately is no object in the runtime that captures
both of these together. Instead, we are forced to separate them into ``*args``
and ``**kwargs``\ , respectively. This means we need to be able to split apart
a single ``ParameterSpecification`` into these two components, and then bring
them back together into a call. To do this, we introduce ``TParams.args`` to
represent the tuple of positional arguments in a given call and
``TParams.kwargs`` to represent the corresponding ``Mapping`` of keywords to
values. These operators can only be used together, as the annotated types for
``*args`` and ``**kwargs`` .
.. code-block::
class G(Generic[TParams]):
def foo(*args: TParams.args, **kwargs: TParams.kwargs) -> int: ... # Accepted
def bar(*args: TParams.kwargs, **kwargs: TParams.args) -> int: ... # Rejected
def baz(*args: TParams.args) -> int: ... # Rejected
stored_arguments: TParams.args # Rejected
def bap(x: TParams.args) -> int: ... # Rejected
def bop(*args: List[TParams.args], **kwargs: TParams.kwargs) -> int: ... # Rejected
def foo(
*args: TParams.args, **kwargs: TParams.kwargs
) -> int: # Accepted
...
def bar(
*args: TParams.kwargs, **kwargs: TParams.args
) -> int: # Rejected
...
def baz(*args: TParams.args) -> int: ... # Rejected
stored_arguments: TParams.args # Rejected
def bap(x: TParams.args) -> int: ... # Rejected
def bop(
*args: List[TParams.args], **kwargs: TParams.kwargs
) -> int: # Rejected
...
Because the default kind of parameter in Python (\ ``(x: int)``\ ) may be addressed both positionally and through its name, two valid invocations of a ``(*args: TParams.args, **kwargs: TParams.kwargs)`` function may give different partitions of the same set of parameters. Therefore we need to make sure that these special types are only brought into the world together, and are used together, so that our usage is valid for all possible partitions.
Because the default kind of parameter in Python (\ ``(x: int)``\ ) may be
addressed both positionally and through its name, two valid invocations of
a ``(*args: TParams.args, **kwargs: TParams.kwargs)`` function may give
different partitions of the same set of parameters. Therefore we need to make
sure that these special types are only brought into the world together, and are
used together, so that our usage is valid for all possible partitions.
With those requirements met, we can now take advantage of the unique properties afforded to us by this set up:
With those requirements met, we can now take advantage of the unique properties
afforded to us by this set up:
* Inside the function, ``args`` has the type ``TParams.args``\ , not ``Tuple[TParams.args, ...]`` as would be with a normal annotation (and likewise with the ``**kwargs``\ )
* A function of type ``Callable[TParams, TReturn]`` can be called with ``(*args, **kwargs)`` if and only if ``args`` has the type ``TParams.args`` and ``kwargs`` has the type ``TParams.kwargs``\ , and that those types both originated from the same function declaration.
* Inside the function, ``args`` has the type ``TParams.args``\ , not
``Tuple[TParams.args, ...]`` as would be with a normal annotation
(and likewise with the ``**kwargs``\ )
* A function of type ``Callable[TParams, TReturn]`` can be called with
``(*args, **kwargs)`` if and only if ``args`` has the type ``TParams.args``
and ``kwargs`` has the type ``TParams.kwargs``\ , and that those types both
originated from the same function declaration.
* A function declared as
``def inner(*args: TParams.args, **kwargs: TParams.kwargs) -> X``
has type ``Callable[TParams, X]``.
With these three properties, we now have the ability to fully type check parameter preserving decorators.
With these three properties, we now have the ability to fully type check
parameter preserving decorators.
One additional form that we want to support is functions that pass only a subset of their arguments on to another function. To avoid shadowing a named or keyword only argument in the ``ParameterSpecification`` we require that the additional arguments be anonymous arguments that precede the ``*args`` and ``*kwargs``
One additional form that we want to support is functions that pass only a subset
of their arguments on to another function. To avoid shadowing a named or keyword
only argument in the ``ParameterSpecification`` we require that the additional
arguments be anonymous arguments that precede the ``*args`` and ``*kwargs``
.. code-block::
@ -182,12 +275,21 @@ One additional form that we want to support is functions that pass only a subset
Backwards Compatibility
-----------------------
The only changes necessary to existing features in ``typing`` is allowing these ``ParameterSpecification`` objects to be the first parameter to ``Callable`` and to be a parameter to ``Generic``. Currently ``Callable`` expects a list of types there and ``Generic`` expects single types, so they are currently mutually exclusive. Otherwise, existing code that doesn't reference the new interfaces will be unaffected.
The only changes necessary to existing features in ``typing`` is allowing these
``ParameterSpecification`` objects to be the first parameter to ``Callable`` and
to be a parameter to ``Generic``. Currently ``Callable`` expects a list of types
there and ``Generic`` expects single types, so they are currently mutually
exclusive. Otherwise, existing code that doesn't reference the new interfaces
will be unaffected.
Reference Implementation
------------------------
The `Pyre <https://pyre-check.org/>`_ type checker supports ``ParameterSpecification``\ s, ``.args`` and ``.kwargs`` in the context of functions. Support for use with ``Generic`` is not yet implemented. A reference implementation of the runtime components needed for those uses is provided in the ``pyre_extensions`` module.
The `Pyre <https://pyre-check.org/>`_ type checker supports
``ParameterSpecification``\ s, ``.args`` and ``.kwargs`` in the context of
functions. Support for use with ``Generic`` is not yet implemented. A reference
implementation of the runtime components needed for those uses is provided in
the ``pyre_extensions`` module.
Rejected Alternatives
---------------------
@ -195,7 +297,9 @@ Rejected Alternatives
Using List Variadics and Map Variadics
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
We considered just trying to make something like this with a callback protocol which was parameterized on a list-type variadic, and a map-type variadic like so:
We considered just trying to make something like this with a callback protocol
which was parameterized on a list-type variadic, and a map-type variadic like
so:
.. code-block::
@ -205,7 +309,9 @@ We considered just trying to make something like this with a callback protocol w
class BetterCallable(typing.Protocol[Tpositionals, Tkeywords, Treturn]):
def __call__(*args: Tpositionals, **kwargs: Tkeywords) -> Treturn: ...
However there are some problems with trying to come up with a consistent solution for those type variables for a given callable. This problem comes up with even the simplest of callables:
However there are some problems with trying to come up with a consistent
solution for those type variables for a given callable. This problem comes up
with even the simplest of callables:
.. code-block::
@ -214,7 +320,10 @@ However there are some problems with trying to come up with a consistent solutio
simple <: BetterCallable[[], {“x”: int}, None]
BetterCallable[[int], [], None] </: BetterCallable[[], {“x”: int}, None]
Any time where a type can implement a protocol in more than one way that aren’t mutually compatible, we can run into situations where we lose information. If we were to make a decorator using this protocol, we have to pick one calling convention to prefer.
Any time where a type can implement a protocol in more than one way that aren’t
mutually compatible, we can run into situations where we lose information. If we
were to make a decorator using this protocol, we have to pick one calling
convention to prefer.
.. code-block::
@ -233,26 +342,46 @@ Any time where a type can implement a protocol in more than one way that aren’
foo(7) # fails under option B
foo(x=7) # fails under option A
The core problem here is that, by default, parameters in Python can either be passed in positionally or as a keyword parameter. This means we really have three categories (positional-only, positional-or-keyword, keyword-only) we’re trying to jam into two categories. This is the same problem that we briefly mentioned when discussing ``.args`` and ``.kwargs``. Fundamentally, in order to capture two categories when there are some things that can be in either category, we need a higher level primitive (\ ``ParameterSpecification``\ ) to capture all three, and then split them out afterward.
The core problem here is that, by default, parameters in Python can either be
passed in positionally or as a keyword parameter. This means we really have
three categories (positional-only, positional-or-keyword, keyword-only) we’re
trying to jam into two categories. This is the same problem that we briefly
mentioned when discussing ``.args`` and ``.kwargs``. Fundamentally, in order to
capture two categories when there are some things that can be in either
category, we need a higher level primitive (\ ``ParameterSpecification``\ ) to
capture all three, and then split them out afterward.
Mutations on ParameterSpecifications
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There are still a class of decorators still not supported with these features: those that mutate (add/remove/change) the parameters of the given function. Defining operators that do these mutations becomes very complicated very quickly, as you have to deal with name collision issues much more prominently. We will defer that work until there is significant demand, and then we would be open to revisiting it.
There are still a class of decorators still not supported with these features:
those that mutate (add/remove/change) the parameters of the given function.
Defining operators that do these mutations becomes very complicated very
quickly, as you have to deal with name collision issues much more prominently.
We will defer that work until there is significant demand, and then we would be
open to revisiting it.
Naming this an ``ArgSpec``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
We think that calling this a ParameterSpecification is more correct than referring to it as an Argument Specification, since callables have parameters, which are distinct from the arguments which are passed to them in a given call site. A given binding for a ParameterSpecification is a set of function parameters, not a call-site’s arguments.
We think that calling this a ParameterSpecification is more correct than
referring to it as an Argument Specification, since callables have parameters,
which are distinct from the arguments which are passed to them in a given call
site. A given binding for a ParameterSpecification is a set of function
parameters, not a call-site’s arguments.
Acknowledgements
----------------
Thanks to all of the members of the Pyre team for their comments on early drafts of this PEP, and for their help with the reference implementation.
Thanks to all of the members of the Pyre team for their comments on early drafts
of this PEP, and for their help with the reference implementation.
Thanks are also due to the whole Python typing community for their early feedback on this idea at a Python typing meetup, leading directly to the much more compact ``.args``\ /\ ``.kwargs`` syntax.
Thanks are also due to the whole Python typing community for their early
feedback on this idea at a Python typing meetup, leading directly to the much
more compact ``.args``\ /\ ``.kwargs`` syntax.
Copyright
---------
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
This document is placed in the public domain or under the CC0-1.0-Universal
license, whichever is more permissive.