855 lines
30 KiB
ReStructuredText
855 lines
30 KiB
ReStructuredText
|
PEP: 677
|
||
|
Title: Callable Type Syntax
|
||
|
Author: Steven Troxler <steven.troxler@gmail.com>,
|
||
|
Pradeep Kumar Srinivasan <gohanpra@gmail.com>
|
||
|
Sponsor: Guido van Rossum <guido at python.org>
|
||
|
Status: Draft
|
||
|
Type: Standards Track
|
||
|
Content-Type: text/x-rst
|
||
|
Created: 13-Dec-2021
|
||
|
Python-Version: 3.11
|
||
|
Post-History:
|
||
|
|
||
|
Abstract
|
||
|
========
|
||
|
|
||
|
This PEP introduces a concise and friendly syntax for callable types,
|
||
|
supporting the same functionality as ``typing.Callable`` but with an
|
||
|
arrow syntax inspired by the syntax for typed function
|
||
|
signatures. This allows types like ``Callable[[int, str], bool]`` to
|
||
|
be written ``(int, str) -> bool``.
|
||
|
|
||
|
The proposed syntax supports all the functionality provided by
|
||
|
``typing.Callable`` and ``typing.Concatenate``, and is intended to
|
||
|
work as a drop-in replacement.
|
||
|
|
||
|
|
||
|
Motivation
|
||
|
==========
|
||
|
|
||
|
Describing Callable Signatures with ``typing.Callable``
|
||
|
-------------------------------------------------------
|
||
|
|
||
|
One way to make code safer and easier to analyze is by making sure
|
||
|
that functions and classes are well-typed. In Python we have type
|
||
|
annotations defined by PEP 484 to provide type hints that can find
|
||
|
bugs as well as helping with editor tooling like tab completion,
|
||
|
static analysis tooling, and code review.
|
||
|
|
||
|
One of the types `defined by PEP 484
|
||
|
<https://www.python.org/dev/peps/pep-0484/#callable>`_ is
|
||
|
``typing.Callable``, which describes a callable value such as a
|
||
|
function or a method. It takes two parameters as inputs but with the
|
||
|
first parameter being either a placeholder like ``...`` or a list of
|
||
|
types. For example:
|
||
|
|
||
|
- ``Callable[..., int]`` indicates a function with arbitrary
|
||
|
parameters returning an integer.
|
||
|
- ``Callable[[str, int], bool]`` indicates a function taking two
|
||
|
positional parameters of types ``str`` and ``int`` and returning a
|
||
|
``bool``.
|
||
|
|
||
|
Of the types defined by PEP 484, ``typing.Callable`` is the most
|
||
|
complex because it is the only one that requires two levels of
|
||
|
brackets in the same type. PEP 612 added ``typing.ParamSpec`` and
|
||
|
``typing.Concatenate`` to help describe functions that pass ``*args``
|
||
|
and ``**kwargs`` to callbacks, which is very common with
|
||
|
decorators. This made ``typing.Callable`` more powerful but also more
|
||
|
complicated.
|
||
|
|
||
|
Problems with ``typing.Callable``
|
||
|
---------------------------------
|
||
|
|
||
|
Empirically `we have found
|
||
|
<https://github.com/pradeep90/annotation_collector#typed-projects---callable-type>`_
|
||
|
that it is common for library authors to make use of untyped or
|
||
|
partially-typed callables (e.g. ``Callable[..., Any]`` or a bare
|
||
|
``Callable``) which we believe is partially a result of the existing
|
||
|
types being hard to use. But using this kind of partially-typed
|
||
|
callable can negate the benefits of static typing. For example,
|
||
|
consider the following code::
|
||
|
|
||
|
from typing import Any, Callable
|
||
|
|
||
|
def with_retries(
|
||
|
f: Callable[..., Any]
|
||
|
) -> Callable[..., Any]:
|
||
|
def wrapper(retry_once, *args, **kwargs):
|
||
|
if retry_once:
|
||
|
try: return f(*args, **kwargs)
|
||
|
except Exception: pass
|
||
|
return f(*args, **kwargs)
|
||
|
return wrapper
|
||
|
|
||
|
@with_retries
|
||
|
def f(x: int) -> int:
|
||
|
return x
|
||
|
|
||
|
|
||
|
f(True, z=15)
|
||
|
|
||
|
The decorator above is clearly not intended to modify the type of the
|
||
|
function it wraps, but because it uses ``Callable[..., Any]`` it
|
||
|
accidentally eliminates the annotations on ``f``, and type checkers
|
||
|
will accept the code above even though it is sure to crash at
|
||
|
runtime. A correct version of this code would look like this::
|
||
|
|
||
|
from typing import Any, Callable, Concatenate, ParamSpec, TypeVar
|
||
|
|
||
|
R = TypeVar("R")
|
||
|
P = ParamSpec("P")
|
||
|
|
||
|
def with_retries(
|
||
|
f: Callable[P, R]
|
||
|
) -> Callable[Concatenate[bool, P] R]:
|
||
|
def wrapper(retry_once: bool, *args: P.args, **kwargs: P.kwargs):
|
||
|
...
|
||
|
return wrapper
|
||
|
|
||
|
...
|
||
|
|
||
|
With these changes, the incorrect attempt to pass ``z`` to ``f``
|
||
|
produces a typecheck error as we would like.
|
||
|
|
||
|
Four usability problems with the way ``typing.Callable`` is
|
||
|
represented may explain why library authors often do not use its full
|
||
|
power:
|
||
|
|
||
|
- It is verbose, particularly for more complex function signatures.
|
||
|
- It does not visually represent the way function headers are written,
|
||
|
which can make it harder to learn and use.
|
||
|
- It requires an explicit import, something we no longer require for
|
||
|
most of the other very common types after PEP 604 (``|`` for
|
||
|
``Union`` types) and PEP 585 (generic collections)
|
||
|
- It relies on two levels of nested square brackets. This can be quite
|
||
|
hard to read, especially when the function arguments themselves have
|
||
|
square brackets.
|
||
|
|
||
|
With our proposed syntax, the properly-typed decorator example becomes
|
||
|
concise and the type representations are visually descriptive::
|
||
|
|
||
|
from typing import Any, ParamSpec, TypeVar
|
||
|
|
||
|
R = TypeVar("R")
|
||
|
P = ParamSpec("P")
|
||
|
|
||
|
def with_retries(
|
||
|
f: (**P) -> R
|
||
|
) -> (bool, **P) -> R:
|
||
|
...
|
||
|
|
||
|
An Arrow Syntax for Callable Types
|
||
|
----------------------------------
|
||
|
|
||
|
We are proposing a succinct, easy-to-use syntax for
|
||
|
``typing.Callable`` that looks similar to function headers in Python.
|
||
|
Our proposal closely follows syntax used by several popular languages
|
||
|
such as `Typescript
|
||
|
<https://basarat.gitbook.io/typescript/type-system/callable#arrow-syntax>`_,
|
||
|
`Kotlin <https://kotlinlang.org/docs/lambdas.html>`_, and `Scala
|
||
|
<https://docs.scala-lang.org/tour/higher-order-functions.html>`_.
|
||
|
|
||
|
Our goals are that:
|
||
|
|
||
|
- Callable types using this syntax will be easier to learn and use,
|
||
|
particularly for developers with experience in other languages.
|
||
|
- Library authors will be more likely to use expressive types for
|
||
|
callables that enable type checkers to better understand code and
|
||
|
find bugs, as in the ``decorator`` example above.
|
||
|
|
||
|
Consider this simplified real-world example from a web server, written
|
||
|
using the existing ``typing.Callable``::
|
||
|
|
||
|
from typing import Awaitable, Callable
|
||
|
from app_logic import Response, UserSetting
|
||
|
|
||
|
|
||
|
async def customize_response_for_settings(
|
||
|
response: Response,
|
||
|
customizer: Callable[[Response, list[UserSetting]], Awaitable[Response]]
|
||
|
) -> Response:
|
||
|
...
|
||
|
|
||
|
With our proposal, this code can be abbreviated to::
|
||
|
|
||
|
from app_logic import Response, UserSetting
|
||
|
|
||
|
def make_endpoint(
|
||
|
response: Response,
|
||
|
customizer: async (Response, list[UserSetting]) -> Response,
|
||
|
) -> Response:
|
||
|
...
|
||
|
|
||
|
This is shorter and requires fewer imports. It also has far less
|
||
|
nesting of square brackets - only one level, as opposed to three in
|
||
|
the original code.
|
||
|
|
||
|
Rationale
|
||
|
=========
|
||
|
|
||
|
The ``Callable`` type is widely used. For example, `as of October 2021
|
||
|
it was
|
||
|
<https://github.com/pradeep90/annotation_collector#overall-stats-in-typeshed>`_
|
||
|
the fifth most common complex type in typeshed, after ``Optional``,
|
||
|
``Tuple``, ``Union``, and ``List``.
|
||
|
|
||
|
Most of the other commonly used types have had their syntax improved
|
||
|
via either PEP 604 or PEP 585. ``Callable`` is used heavily enough to
|
||
|
similarly justify a more usable syntax.
|
||
|
|
||
|
In this proposal, we chose to support all the existing semantics of
|
||
|
``typing.Callable``, without adding support for new features. We took
|
||
|
this decision after examining how frequently each feature might be
|
||
|
used in existing typed and untyped open-source code. We determined
|
||
|
that the vast majority of use cases are covered.
|
||
|
|
||
|
We considered adding support for named, optional, and variadic
|
||
|
arguments. However, we decided against including these features, as
|
||
|
our analysis showed they are infrequently used. When they are really
|
||
|
needed, it is possible to type these using `Callback Protocols
|
||
|
<https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols>`_.
|
||
|
|
||
|
See the Rejected Alternatives section for more detailed discussion
|
||
|
about omitted features.
|
||
|
|
||
|
Specification
|
||
|
=============
|
||
|
|
||
|
Typing Behavior
|
||
|
---------------
|
||
|
|
||
|
Type checkers should treat the new syntax with exactly the same
|
||
|
semantics as ``typing.Callable``.
|
||
|
|
||
|
As such, a type checker should treat the following pairs exactly the
|
||
|
same::
|
||
|
|
||
|
from typing import Awaitable, Callable, Concatenate, ParamSpec, TypeVarTuple
|
||
|
|
||
|
P = ParamSpec("P")
|
||
|
Ts = TypeVarTuple('Ts')
|
||
|
|
||
|
f0: () -> bool
|
||
|
f0: Callable[[], bool]
|
||
|
|
||
|
f1: (int, str) -> bool
|
||
|
f1: Callable[[int, str], bool]
|
||
|
|
||
|
f2: (...) -> bool
|
||
|
f2: Callable[..., bool]
|
||
|
|
||
|
f3: async (str) -> str
|
||
|
f3: Callable[[str], Awaitable[str]]
|
||
|
|
||
|
f4: (**P) -> bool
|
||
|
f4: Callable[P, bool]
|
||
|
|
||
|
f5: (int, **P) -> bool
|
||
|
f5: Callable[Concatenate[int, P], bool]
|
||
|
|
||
|
f6: (*Ts) -> bool
|
||
|
f6: Callable[[*Ts], bool]
|
||
|
|
||
|
f7: (int, *Ts, str) -> bool
|
||
|
f7: Callable[[int, *Ts, str], bool]
|
||
|
|
||
|
|
||
|
Grammar and AST
|
||
|
---------------
|
||
|
|
||
|
The proposed new syntax can be described by these AST changes ::
|
||
|
|
||
|
expr = <prexisting_expr_kinds>
|
||
|
| AsyncCallableType(callable_type_arguments args, expr returns)
|
||
|
| CallableType(callable_type_arguments args, expr returns)
|
||
|
|
||
|
callable_type_arguments = AnyArguments
|
||
|
| ArgumentsList(expr* posonlyargs)
|
||
|
| Concatenation(expr* posonlyargs, expr param_spec)
|
||
|
|
||
|
|
||
|
Here are our proposed changes to the `Python Grammar
|
||
|
<https://docs.python.org/3/reference/grammar.htm>`::
|
||
|
|
||
|
expression:
|
||
|
| disjunction disjunction 'else' expression
|
||
|
| callable_type_expression
|
||
|
| disjunction
|
||
|
| lambdef
|
||
|
|
||
|
callable_type_expression:
|
||
|
| callable_type_arguments '->' expression
|
||
|
| ASYNC callable_type_arguments '->' expression
|
||
|
|
||
|
callable_type_arguments:
|
||
|
| '(' '...' [','] ')'
|
||
|
| '(' callable_type_positional_argument* ')'
|
||
|
| '(' callable_type_positional_argument* callable_type_param_spec ')'
|
||
|
|
||
|
callable_type_positional_argument:
|
||
|
| !'...' expression ','
|
||
|
| !'...' expression &')'
|
||
|
|
||
|
callable_type_param_spec:
|
||
|
| '**' expression ','
|
||
|
| '**' expression &')'
|
||
|
|
||
|
|
||
|
|
||
|
If PEP 646 is accepted, we intend to include support for unpacked
|
||
|
types by modifying the grammar for
|
||
|
``callable_type_positional_argument`` as follows::
|
||
|
|
||
|
callable_type_positional_argument:
|
||
|
| expression ','
|
||
|
| expression &')'
|
||
|
| '*' expression ','
|
||
|
| '*' expression &')'
|
||
|
|
||
|
|
||
|
Implications of the Grammar
|
||
|
---------------------------
|
||
|
|
||
|
|
||
|
Precedence of ->
|
||
|
~~~~~~~~~~~~~~~~
|
||
|
|
||
|
|
||
|
``->`` binds less tightly than other operators, both inside types and
|
||
|
in function signatures, so the following two callable types are
|
||
|
equivalent::
|
||
|
|
||
|
(int) -> str | bool
|
||
|
(int) -> (str | bool)
|
||
|
|
||
|
|
||
|
``->`` associates to the right, both inside types and in function
|
||
|
signatures. So the following pairs are equivalent::
|
||
|
|
||
|
(int) -> (str) -> bool
|
||
|
(int) -> ((str) -> bool)
|
||
|
|
||
|
def f() -> (int, str) -> bool: pass
|
||
|
def f() -> ((int, str) -> bool): pass
|
||
|
|
||
|
def f() -> (int) -> (str) -> bool: pass
|
||
|
def f() -> ((int) -> ((str) -> bool)): pass
|
||
|
|
||
|
|
||
|
Because operators bind more tightly than ``->``, parentheses are
|
||
|
required whenever an arrow type is intended to be inside an argument
|
||
|
to an operator like ``|``::
|
||
|
|
||
|
(int) -> bool | () -> bool # syntax error!
|
||
|
(int) -> bool | (() -> bool) # okay
|
||
|
|
||
|
|
||
|
We discussed each of these behaviors and believe they are desirable:
|
||
|
|
||
|
- Union types (represented by ``A | B`` according to PEP 604) are
|
||
|
valid in function signature returns, so we need to allow operators
|
||
|
in the return position for consistency.
|
||
|
- Given that operators bind more tightly than ``->`` it is correct
|
||
|
that a type like ```bool | () -> bool`` must be a syntax error. We
|
||
|
should be sure the error message is clear because this may be a
|
||
|
common mistake.
|
||
|
- Associating ``->`` to the right, rather than requiring explicit
|
||
|
parentheses, is consistent with other languages like TypeScript and
|
||
|
respects the principle that valid expressions should normally be
|
||
|
substitutable when possible.
|
||
|
|
||
|
``async`` Keyword
|
||
|
~~~~~~~~~~~~~~~~~
|
||
|
|
||
|
All of the binding rules still work for async callable types::
|
||
|
|
||
|
(int) -> async (float) -> str | bool
|
||
|
(int) -> (async (float) -> (str | bool))
|
||
|
|
||
|
def f() -> async (int, str) -> bool: pass
|
||
|
def f() -> (async (int, str) -> bool): pass
|
||
|
|
||
|
def f() -> async (int) -> async (str) -> bool: pass
|
||
|
def f() -> (async (int) -> (async (str) -> bool)): pass
|
||
|
|
||
|
|
||
|
Trailing Commas
|
||
|
~~~~~~~~~~~~~~~
|
||
|
|
||
|
- Following the precedent of function signatures, putting a comma in
|
||
|
an empty arguments list is illegal, ``(,) -> bool`` is a syntax
|
||
|
error.
|
||
|
- Again following precedent, trailing commas are otherwise always
|
||
|
permitted::
|
||
|
|
||
|
|
||
|
((int,) -> bool == (int) -> bool
|
||
|
((int, **P,) -> bool == (int, **P) -> bool
|
||
|
((...,) -> bool) == ((...) -> bool)
|
||
|
|
||
|
Allowing trailing commas also gives autoformatters more flexibility
|
||
|
when splitting callable types across lines, which is always legal
|
||
|
following standard python whitespace rules.
|
||
|
|
||
|
|
||
|
Disallowing ``...`` as an Argument Type
|
||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
|
||
|
Under normal circumstances, any valid expression is permitted where we
|
||
|
want a type annotation and ``...`` is a valid expression. This is
|
||
|
never semantically valid and all type checkers would reject it, but
|
||
|
the grammar would allow it if we did not explicitly prevent this.
|
||
|
|
||
|
We decided that there were compelling reasons to prevent it: - The
|
||
|
semantics of ``(...) -> bool`` are different from ``(T) -> bool`` for
|
||
|
any valid type T: ``(...)`` is a special form indicating
|
||
|
``AnyArguments`` whereas ``T`` is a type parameter in the arguments
|
||
|
list. - ``...`` is used as a placeholder default value to indicate an
|
||
|
optional argument in stubs and Callback Protocols. Allowing it in the
|
||
|
position of a type could easily lead to confusion and possibly bugs
|
||
|
due to typos.
|
||
|
|
||
|
Since ``...`` is meaningless as a type and there are usability
|
||
|
concerns, our grammar rules it out and the following is a syntax
|
||
|
error::
|
||
|
|
||
|
(int, ...) -> bool
|
||
|
|
||
|
Incompatibility with other possible uses of ``*`` and ``**``
|
||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
|
||
|
The use of ``**P`` for supporting PEP 612 ``ParamSpec`` rules out any
|
||
|
future proposal using a bare ``**<some_type>`` to type
|
||
|
``kwargs``. This seems acceptable because:
|
||
|
|
||
|
- If we ever do want such a syntax, it would be clearer to require an
|
||
|
argument name anyway. This would also make the type look more
|
||
|
similar to a function signature. In other words, if we ever support
|
||
|
typing ``kwargs`` in callable types, we would prefer ``(int,
|
||
|
**kwargs: str)`` rather than ``(int, **str)``.
|
||
|
- PEP 646 unpacking syntax would rule out using ``*<some_type>`` for
|
||
|
``args``. The ``kwargs`` case is similar enough that this rules out
|
||
|
a bare ``**<some_type>`` anyway.
|
||
|
|
||
|
|
||
|
|
||
|
Compatibility with Arrow-Based Lambda Syntax
|
||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
|
||
|
To the best of our knowledge there is no active discussion of
|
||
|
arrow-style lambda syntax that we are aware of, but it is nonetheless
|
||
|
worth considering what possibilities would be ruled out by adopting
|
||
|
this proposal.
|
||
|
|
||
|
It would be incompatible with this proposal to adopt the same a
|
||
|
parenthesized ``->``-based arrow syntax for lambdas, e.g. ``(x, y) ->
|
||
|
x + y`` for ``lambda x, y: x + y``.
|
||
|
|
||
|
|
||
|
Our view is that if we want arrow syntax for lambdas in the future, it
|
||
|
would be a better choice to use ``=>``, e.g. ``(x, y) => x + y``.
|
||
|
Many languages use the same arrow token for both lambdas and callable
|
||
|
types, but Python is unique in that types are expressions and have to
|
||
|
evaluate to runtime values. Our view is that this merits using
|
||
|
separate tokens, and given the existing use of ``->`` for return types
|
||
|
in function signatures it would be more coherent to use ``->`` for
|
||
|
callable types and ``=>`` for lambdas.
|
||
|
|
||
|
Runtime Behavior
|
||
|
----------------
|
||
|
|
||
|
Our tentative plan is that:
|
||
|
|
||
|
- The ``__repr__`` will show an arrow syntax literal.
|
||
|
- We will provide a new API where the runtime data structure can be
|
||
|
accessed in the same manner as the AST data structure.
|
||
|
- We will ensure that we provide an API that is backward-compatible
|
||
|
with ``typing.Callable`` and ``typing.Concatenate``, specifically
|
||
|
the behavior of ``__args__`` and ``__parameters__``.
|
||
|
|
||
|
Because these details are still under debate we are currently
|
||
|
maintaining `a separate doc
|
||
|
<https://docs.google.com/document/d/15nmTDA_39Lo-EULQQwdwYx_Q1IYX4dD5WPnHbFG71Lk/edit>`_
|
||
|
with details about the new builtins, the evaluation model, how to
|
||
|
provide both a backward-compatible and more structured API, and
|
||
|
possible alternatives to the current plan.
|
||
|
|
||
|
Once the plan is finalized we will include a full specification of
|
||
|
runtime behavior in this section of the PEP.
|
||
|
|
||
|
Rejected Alternatives
|
||
|
=====================
|
||
|
|
||
|
Many of the alternatives we considered would have been more expressive
|
||
|
than ``typing.Callable``, for example adding support for describing
|
||
|
signatures that include named, optional, and variadic arguments.
|
||
|
|
||
|
To determine which features we most needed to support with a callable
|
||
|
type syntax, we did an extensive analysis of existing projects:
|
||
|
|
||
|
- `stats on the use of the Callable type <https://github.com/pradeep90/annotation_collector#typed-projects---callable-type>`_;
|
||
|
- `stats on how untyped and partially-typed callbacks are actually used <https://github.com/pradeep90/annotation_collector#typed-projects---callback-usage>`_.
|
||
|
|
||
|
We decided on a simple proposal with improved syntax for the existing
|
||
|
``Callable`` type because the vast majority of callbacks can be correctly
|
||
|
described by the existing ``typing.Callable`` semantics:
|
||
|
|
||
|
- Positional parameters: By far the most important case to handle well
|
||
|
is simple callable types with positional parameters, such as
|
||
|
``(int, str) -> bool``
|
||
|
- ParamSpec and Concatenate: The next most important feature is good
|
||
|
support for PEP 612 ``ParamSpec`` and ``Concatenate`` types like
|
||
|
``(**P) -> bool`` and ``(int, **P) -> bool``. These are common
|
||
|
primarily because of the heavy use of decorator patterns in python
|
||
|
code.
|
||
|
- TypeVarTuples: The next most important feature, assuming PEP 646 is
|
||
|
accepted, is for unpacked types which are common because of cases
|
||
|
where a wrapper passes along ``*args`` to some other function.
|
||
|
|
||
|
Features that other, more complicated proposals would support account
|
||
|
for fewer than 2% of the use cases we found. These are already
|
||
|
expressible using Callback Protocols, and since they are uncommon we
|
||
|
decided that it made more sense to move forward with a simpler syntax.
|
||
|
|
||
|
Extended Syntax Supporting Named and Optional Arguments
|
||
|
-------------------------------------------------------
|
||
|
|
||
|
Another alternative was for a compatible but more complex syntax that
|
||
|
could express everything in this PEP but also named, optional, and
|
||
|
variadic arguments. In this “extended” syntax proposal the following
|
||
|
types would have been equivalent::
|
||
|
|
||
|
class Function(typing.Protocol):
|
||
|
def f(self, x: int, /, y: float, *, z: bool = ..., **kwargs: str) -> bool:
|
||
|
...
|
||
|
|
||
|
Function = (int, y: float, *, z: bool = ..., **kwargs: str) -> bool
|
||
|
|
||
|
Advantages of this syntax include: - Most of the advantages of the
|
||
|
proposal in this PEP (conciseness, PEP 612 support, etc) -
|
||
|
Furthermore, the ability to handle named, optional, and variadic
|
||
|
arguments
|
||
|
|
||
|
We decided against proposing it for the following reasons:
|
||
|
|
||
|
- The implementation would have been more difficult, and usage stats
|
||
|
demonstrate that fewer than 3% of use cases would benefit from any
|
||
|
of the added features.
|
||
|
- The group that debated these proposals was split down the middle
|
||
|
about whether these changes are even desirable:
|
||
|
|
||
|
- On the one hand, they make callable types more expressive. On the
|
||
|
other hand, they could easily confuse users who have not read the
|
||
|
full specification of callable type syntax.
|
||
|
- We believe the simpler syntax proposed in this PEP, which
|
||
|
introduces no new semantics and closely mimics syntax in other
|
||
|
popular languages like Kotlin, Scala, and TypesScript, is much
|
||
|
less likely to confuse users.
|
||
|
|
||
|
- We intend to implement the current proposal in a way that is
|
||
|
forward-compatible with the more complicated extended syntax. If the
|
||
|
community decides after more experience and discussion that we want
|
||
|
the additional features, they should be straightforward to propose
|
||
|
in the future.
|
||
|
- We realized that because of overloads, it is not possible to replace
|
||
|
all need for Callback Protocols even with an extended syntax. This
|
||
|
makes us prefer proposing a simple solution that handles most use
|
||
|
cases well.
|
||
|
|
||
|
We confirmed that the current proposal is forward-compatible with
|
||
|
extended syntax by
|
||
|
`implementing <https://github.com/stroxler/cpython/tree/callable-type-syntax--extended>`_
|
||
|
a quick-and-dirty grammar and AST on top of the grammar and AST for
|
||
|
the current proposal.
|
||
|
|
||
|
|
||
|
Syntax Closer to Function Signatures
|
||
|
------------------------------------
|
||
|
|
||
|
One alternative we had floated was a syntax much more similar to
|
||
|
function signatures.
|
||
|
|
||
|
In this proposal, the following types would have been equivalent::
|
||
|
|
||
|
class Function(typing.Protocol):
|
||
|
def f(self, x: int, /, y: float, *, z: bool = ..., **kwargs: str) -> bool:
|
||
|
...
|
||
|
|
||
|
Function = (x: int, /, y: float, *, z: bool = ..., **kwargs: str) -> bool
|
||
|
|
||
|
|
||
|
The benefits of this proposal would have included:
|
||
|
|
||
|
- Perfect syntactic consistency between signatures and callable types.
|
||
|
- Support for more features of function signatures (named, optional,
|
||
|
variadic args) that this PEP does not support.
|
||
|
|
||
|
Key downsides that led us to reject the idea include the following:
|
||
|
|
||
|
- A large majority of use cases only use positional-only arguments,
|
||
|
and this syntax would be more verbose for that use case, both
|
||
|
because of requiring argument names and an explicit ``/``, for
|
||
|
example ``(int, /) -> bool`` where our proposal allows ``(int) ->
|
||
|
bool``
|
||
|
- The requirement for explicit ``/`` for positional-only arguments has
|
||
|
a high risk of causing frequent bugs - which often would not be
|
||
|
detected by unit tests - where library authors would accidentally
|
||
|
use types with named arguments.
|
||
|
- Our analysis suggests that support for ``ParamSpec`` is key, but the
|
||
|
scoping rules laid out in PEP 612 would have made this difficult.
|
||
|
|
||
|
|
||
|
Other Proposals Considered
|
||
|
--------------------------
|
||
|
|
||
|
Functions-as-Types
|
||
|
~~~~~~~~~~~~~~~~~~
|
||
|
|
||
|
An idea we looked at very early on was to `allow using functions as
|
||
|
types
|
||
|
<https://docs.google.com/document/d/1rv6CCDnmLIeDrYlXe-QcyT0xNPSYAuO1EBYjU3imU5s/edit?usp=sharing>`_.
|
||
|
The idea is allowing a function to stand in for its own call
|
||
|
signature, with roughly the same semantics as the ``__call__`` method
|
||
|
of Callback Protocols. Think this may be a great idea and worth its
|
||
|
own PEP, but that it is not a good alternative to improving the
|
||
|
usability of callable types:
|
||
|
|
||
|
- Using functions as types would not give us a new way of describing
|
||
|
function types as first class values. Instead, they would require a
|
||
|
function definition statement that effectively defines a type alias
|
||
|
(much as a Callable Protocol class statement does).
|
||
|
- Functions-as-types would support almost exactly the same features
|
||
|
that Callable Protocols do today: named, optional, and variadic args
|
||
|
as well as the ability to define overloads.
|
||
|
|
||
|
Another reason we don't view functions-as-types as a good alternative
|
||
|
is that it would be difficult to handle ``ParamSpec``, which we
|
||
|
consider a critical feature to support.
|
||
|
|
||
|
Parenthesis-Free Syntax
|
||
|
~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
|
||
|
We considered a parentheses-free syntax that would have been even more
|
||
|
concise::
|
||
|
|
||
|
int, str -> bool
|
||
|
|
||
|
We decided against it because this is not visually as similar to
|
||
|
existing function header syntax. Moreover, it is visually similar to
|
||
|
lambdas, which bind names with no parentheses: ``lambda x, y: x ==
|
||
|
y``.
|
||
|
|
||
|
Introducing type-strings
|
||
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
|
||
|
Another idea was adding a new “special string” syntax and putting the type
|
||
|
inside of it, for example ``t”(int, str) -> bool”``. We rejected this
|
||
|
because it is not as readable, and seems out of step with `guidance
|
||
|
<https://mail.python.org/archives/list/python-dev@python.org/message/SZLWVYV2HPLU6AH7DOUD7DWFUGBJGQAY/>`_
|
||
|
from the Steering Council on ensuring that type expressions do not
|
||
|
diverge from the rest of Python's syntax.
|
||
|
|
||
|
|
||
|
Backward Compatibility
|
||
|
======================
|
||
|
|
||
|
This PEP proposes a major syntax improvement over ``typing.Callable``,
|
||
|
but the static semantics are the same.
|
||
|
|
||
|
As such, the only thing we need for backward compatibility is to
|
||
|
ensure that types specified via the new syntax behave the same as
|
||
|
equivalent ``typing.Callable`` and ``typing.Concatenate`` values they
|
||
|
intend to replace.
|
||
|
|
||
|
There is no particular interaction between this proposal and ``from
|
||
|
__future__ import annotations`` - just like any other type annotation
|
||
|
it will be unparsed to a string at module import, and
|
||
|
``typing.get_type_hints`` should correctly evaluate the resulting
|
||
|
strings in cases where that is possible.
|
||
|
|
||
|
This is discussed in more detail in the Runtime Behavior section.
|
||
|
|
||
|
|
||
|
Reference Implementation
|
||
|
========================
|
||
|
|
||
|
We have a working `implementation
|
||
|
<https://github.com/stroxler/cpython/tree/callable-type-syntax--shorthand>`_
|
||
|
of the AST and Grammar with tests verifying that the grammar proposed
|
||
|
here has the desired behaviors.
|
||
|
|
||
|
The runtime behavior is not yet implemented. As discussed in the
|
||
|
`Runtime Behavior`_ portion of the spec we have a detailed plan for
|
||
|
both a backward-compatible API and a more structured API in
|
||
|
`a separate doc
|
||
|
<https://docs.google.com/document/d/15nmTDA_39Lo-EULQQwdwYx_Q1IYX4dD5WPnHbFG71Lk/edit>`_
|
||
|
where we are also open to discussion and alternative ideas.
|
||
|
|
||
|
|
||
|
Open Issues
|
||
|
===========
|
||
|
|
||
|
Details of the Runtime API
|
||
|
--------------------------
|
||
|
|
||
|
Once we have finalized all details of the runtime behavior, we
|
||
|
will need to add a full specification of the behavior to the
|
||
|
`Runtime Behavior`_ section of this PEP as well as include that
|
||
|
behavior in our reference implementation.
|
||
|
|
||
|
Optimizing ``SyntaxError`` messages
|
||
|
-----------------------------------
|
||
|
|
||
|
The current reference implementation has a fully-functional parser and
|
||
|
all edge cases presented here have been tested.
|
||
|
|
||
|
But there are some known cases where the errors are not as informative
|
||
|
as we would like. For example, because ``(int, ...) -> bool`` is
|
||
|
illegal but ``(int, ...)`` is a valid tuple, we currently produce a
|
||
|
syntax error flagging the ``->`` as the problem even though the real
|
||
|
cause of the error is using ``...`` as an argument type.
|
||
|
|
||
|
This is not part of the specification *per se* but is an important
|
||
|
detail to address in our implementation. The solution will likely
|
||
|
involve adding ``invalid_.*`` rules to ``python.gram`` and customizing
|
||
|
error messages.
|
||
|
|
||
|
Resources
|
||
|
=========
|
||
|
|
||
|
Background and History
|
||
|
----------------------
|
||
|
|
||
|
`PEP 484 specifies
|
||
|
<https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code>`_
|
||
|
a very similar syntax for function type hint *comments* for use in
|
||
|
code that needs to work on Python 2.7. For example::
|
||
|
|
||
|
def f(x, y):
|
||
|
# type: (int, str) -> bool
|
||
|
...
|
||
|
|
||
|
At that time we used indexing operations to specify generic types like
|
||
|
``typing.Callable`` because we decided not to add syntax for
|
||
|
types. However, we have since begun to do so, e.g. with PEP 604.
|
||
|
|
||
|
**Maggie** proposed better callable type syntax as part of a larger
|
||
|
`presentation on typing simplifications
|
||
|
<https://drive.google.com/file/d/1XhqTKoO6RHtz7zXqW5Wgq9nzaEz9TXjI/view>`_
|
||
|
at the PyCon Typing Summit 2021.
|
||
|
|
||
|
**Steven** `brought up this proposal on typing-sig
|
||
|
<https://mail.python.org/archives/list/typing-sig@python.org/thread/3JNXLYH5VFPBNIVKT6FFBVVFCZO4GFR2>`. We
|
||
|
had several meetings to discuss alternatives, and `this presentation
|
||
|
<https://www.dropbox.com/s/sshgtr4p30cs0vc/Python%20Callable%20Syntax%20Proposals.pdf?dl=0>`_
|
||
|
led us to the current proposal.
|
||
|
|
||
|
**Pradeep** `brought this proposal to python-dev
|
||
|
<https://mail.python.org/archives/list/python-dev@python.org/thread/VBHJOS3LOXGVU6I4FABM6DKHH65GGCUB>`_
|
||
|
for feedback.
|
||
|
|
||
|
Other Languages
|
||
|
---------------
|
||
|
|
||
|
Many popular programming languages use an arrow syntax similar
|
||
|
to the one we are proposing here
|
||
|
|
||
|
the same ``->`` arrow token we are proposing here.
|
||
|
almost identical to the ones we are proposing here
|
||
|
|
||
|
TypeScript
|
||
|
~~~~~~~~~~
|
||
|
|
||
|
In `TypeScript
|
||
|
<https://basarat.gitbook.io/typescript/type-system/callable#arrow-syntax>`_,
|
||
|
function types are expressed in a syntax almost the same as the one we
|
||
|
are proposing, but the arrow token is ``=>`` and arguments have names::
|
||
|
|
||
|
(x: int, y: str) => bool
|
||
|
|
||
|
The names of the arguments are not actually relevant to the type. So,
|
||
|
for example, this is the same callable type::
|
||
|
|
||
|
(a: int, b: str) => bool
|
||
|
|
||
|
Kotlin
|
||
|
~~~~~~
|
||
|
|
||
|
Function types in `Kotlin <https://kotlinlang.org/docs/lambdas.html>`_ permit
|
||
|
an identical syntax to the one we are proposing, for example::
|
||
|
|
||
|
(Int, String) -> Bool
|
||
|
|
||
|
It also optionally allows adding names to the arguments, for example::
|
||
|
|
||
|
(x: Int, y: String) -> Bool
|
||
|
|
||
|
As in TypeScript, the argument names if provided are just there for documentation
|
||
|
and are not part of the type itself.
|
||
|
|
||
|
Scala
|
||
|
~~~~~
|
||
|
|
||
|
`Scala <https://docs.scala-lang.org/tour/higher-order-functions.html>`_
|
||
|
uses the ``=>`` arrow for function types. Other than that, their syntax is
|
||
|
the same as the one we are proposing, for example::
|
||
|
|
||
|
(Int, String) => Bool
|
||
|
|
||
|
Scala, like Python, has the ability to provide function arguments by name.
|
||
|
Funciton types can optionally include names, for example::
|
||
|
|
||
|
(x: Int, y: String) => Bool
|
||
|
|
||
|
Unlike in TypeScript and Kotlin, these names are part of the type if
|
||
|
provided - any function implementing the type must use the same names.
|
||
|
This is similar to the extended syntax proposal we described in our
|
||
|
`Rejected Alternatives`_ section.
|
||
|
|
||
|
The ML Language Family
|
||
|
~~~~~~~~~~~~~~~~~~~~~~
|
||
|
|
||
|
Languages in the ML family, including `F#
|
||
|
<https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/fsharp-types#syntax-for-types>`_,
|
||
|
`OCaml
|
||
|
<https://www2.ocaml.org/learn/tutorials/basics.html#Defining-a-function>`_,
|
||
|
and `Haskell <https://wiki.haskell.org/Type_signature>`_, all use
|
||
|
``->`` to represent function types. All of them use a parentheses-free
|
||
|
syntax with multiple arrows, for example in Haskell::
|
||
|
|
||
|
Integer -> String -> Bool
|
||
|
|
||
|
The use of multiple arrows, which differs from our proposal, makes
|
||
|
sense for languages in this family because they use automatic
|
||
|
`currying <https://en.wikipedia.org/wiki/Currying>`_ of function arguments,
|
||
|
which means that a multi-argument function behaves like a single-argument
|
||
|
function returning a function.
|
||
|
|
||
|
Acknowledgments
|
||
|
---------------
|
||
|
|
||
|
Thanks to the following people for their feedback on the PEP and help
|
||
|
planning the reference implementation:
|
||
|
|
||
|
Alex Waygood, Guido Van Rossum, Eric Traut, James Hilton-Balfe, Maggie
|
||
|
Moss, Shannon Zhu
|
||
|
|
||
|
TODO: MAKE SURE THE THANKS STAYS UP TO DATE
|
||
|
|
||
|
|
||
|
Copyright
|
||
|
=========
|
||
|
|
||
|
This document is placed in the public domain or under the
|
||
|
CC0-1.0-Universal license, whichever is more permissive.
|
||
|
|
||
|
|
||
|
..
|
||
|
Local Variables:
|
||
|
mode: indented-text
|
||
|
indent-tabs-mode: nil
|
||
|
sentence-end-double-space: t
|
||
|
fill-column: 70
|
||
|
coding: utf-8
|
||
|
End:
|