python-peps/pep-0677.rst

PEP: 677
Title: Callable Type Syntax
Author: Steven Troxler <steven.troxler@gmail.com>,
        Pradeep Kumar Srinivasan <gohanpra@gmail.com>
Sponsor: Guido van Rossum <guido at python.org>
Discussions-To: python-dev@python.org
Status: Rejected
Type: Standards Track
Topic: Typing
Content-Type: text/x-rst
Created: 13-Dec-2021
Python-Version: 3.11
Post-History: 16-Dec-2021
Resolution: https://mail.python.org/archives/list/python-dev@python.org/message/NHCLHCU2XCWTBGF732WESMN42YYVKOXB/

Abstract
========

This PEP introduces a concise and friendly syntax for callable types,
supporting the same functionality as ``typing.Callable`` but with an
arrow syntax inspired by the syntax for typed function
signatures. This allows types like ``Callable[[int, str], bool]`` to
be written as ``(int, str) -> bool``.

The proposed syntax supports all the functionality provided by
``typing.Callable`` and ``typing.Concatenate``, and is intended to
work as a drop-in replacement.


Motivation
==========

One way to make code safer and easier to analyze is by making sure
that functions and classes are well-typed.  In Python we have type
annotations, the framework for which is defined in :pep:`484`, to provide
type hints that can find bugs as well as helping with editor tooling
like tab completion, static analysis tooling, and code review.

Consider the following untyped code::

    def flat_map(func, l):
        out = []
        for element in l:
            out.extend(func(element))
        return out


    def wrap(x: int) -> list[int]:
        return [x]

    def add(x: int, y: int) -> int:
        return x + y

    flat_map(wrap, [1, 2, 3])  # no runtime error, output is [1, 2, 3]
    flat_map(add, [1, 2, 3])   # runtime error: `add` expects 2 arguments, got 1


We can add types to this example to detect the runtime error::

    from typing import Callable

    def flat_map(
        func: Callable[[int], list[int]],
        l: list[int]
    ) -> list[int]:
        ....

    ...


    flat_map(wrap, [1, 2, 3])  # type checks okay, output is [1, 2, 3]
    flat_map(add, [1, 2, 3])   # type check error

There are a few usability challenges with ``Callable`` we can see here:

- It is verbose, particularly for more complex function signatures.
- It relies on two levels of nested brackets, unlike any other generic
  type. This can be especially hard to read when some of the type
  parameters are themselves generic types.
- The bracket structure is not visually similar to how function signatures
  are written.
- It requires an explicit import, unlike many of the other most common
  types like ``list`` and ``dict``.

Possibly as a result, `programmers often fail to write complete
Callable types
<https://github.com/pradeep90/annotation_collector#typed-projects---callable-type>`_.
Such untyped or partially-typed callable types do not check the
parameter types or return types of the given callable and thus negate
the benefits of static typing. For example, they might write this::


    from typing import Callable

    def flat_map(
        func: Callable[..., Any],
        l: list[int]
    ) -> list[int]:
        ....

    ...


    flat_map(add, [1, 2, 3])  # oops, no type check error!

There's some partial type information here - we at least know that ``func``
needs to be callable. But we've dropped too much type information for
type checkers to find the bug.

With our proposal, the example looks like this::

    def flat_map(
        func: (int) -> list[int],
        l: list[int]
    ) -> list[int]:
        out = []
        for element in l:
            out.extend(f(element))
        return out

    ...

The type ``(int) -> list[int]`` is more concise, uses an arrow similar
to the one indicating a return type in a function header, avoids
nested brackets, and does not require an import.


Rationale
=========

The ``Callable`` type is widely used. For example, `as of October 2021
it was
<https://github.com/pradeep90/annotation_collector#overall-stats-in-typeshed>`_
the fifth most common complex type in typeshed, after ``Optional``,
``Tuple``, ``Union``, and ``List``.

The others have had their syntax improved and the need for imports
eliminated by either :pep:`604` or :pep:`585`:

- ``typing.Optional[int]`` is written ``int | None``
- ``typing.Union[int, str]`` is written ``int | str``
- ``typing.List[int]`` is written ``list[int]``
- ``typing.Tuple[int, str]`` is written ``tuple[int, str]``

The ``typing.Callable`` type is used almost as often as these other
types, is more complicated to read and write, and still requires an
import and bracket-based syntax.

In this proposal, we chose to support all the existing semantics of
``typing.Callable``, without adding support for new features. We made
this decision after examining how frequently each feature might be
used in existing typed and untyped open-source code. We determined
that the vast majority of use cases are covered.

We considered adding support for named, optional, and variadic
arguments. However, we decided against including these features, as
our analysis showed they are infrequently used. When they are really
needed, it is possible to type these using `callback protocols
<https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols>`_.

An Arrow Syntax for Callable Types
----------------------------------

We are proposing a succinct, easy-to-use syntax for
``typing.Callable`` that looks similar to function headers in Python.
Our proposal closely follows syntax used by several popular languages
such as `Typescript
<https://basarat.gitbook.io/typescript/type-system/callable#arrow-syntax>`_,
`Kotlin <https://kotlinlang.org/docs/lambdas.html>`_, and `Scala
<https://docs.scala-lang.org/tour/higher-order-functions.html>`_.

Our goals are that:

- Callable types using this syntax will be easier to learn and use,
  particularly for developers with experience in other languages.
- Library authors will be more likely to use expressive types for
  callables that enable type checkers to better understand code and
  find bugs, as in the ``decorator`` example above.

Consider this simplified real-world example from a web server, written
using the existing ``typing.Callable``::

    from typing import Awaitable, Callable
    from app_logic import Response, UserSetting


    def customize_response(
        response: Response,
        customizer: Callable[[Response, list[UserSetting]], Awaitable[Response]]
    ) -> Response:
       ...

With our proposal, this code can be abbreviated to::

    from app_logic import Response, UserSetting

    def customize_response(
        response: Response,
        customizer: async (Response, list[UserSetting]) -> Response,
    ) -> Response:
        ...

This is shorter and requires fewer imports. It also has far less
nesting of square brackets - only one level, as opposed to three in
the original code.

Compact Syntax for ``ParamSpec``
--------------------------------

A particularly common case where library authors leave off type information
for callables is when defining decorators. Consider the following::


    from typing import Any, Callable

    def with_retries(
        f: Callable[..., Any]
    ) -> Callable[..., Any]:
        def wrapper(retry_once, *args, **kwargs):
            if retry_once:
                try: return f(*args, **kwargs)
                except Exception: pass
            return f(*args, **kwargs)
        return wrapper

    @with_retries
    def f(x: int) -> int:
        return x


    f(y=10)  # oops - no type error!

In the code above, it is clear that the decorator should produce a
function whose signature is like that of the argument ``f`` other
than an additional ``retry_once`` argument. But the use of ``...``
prevents a type checker from seeing this and alerting a user that
``f(y=10)`` is invalid.


With :pep:`612` it is possible to type decorators like this correctly
as follows::

    from typing import Any, Callable, Concatenate, ParamSpec, TypeVar

    R = TypeVar("R")
    P = ParamSpec("P")

    def with_retries(
        f: Callable[P, R]
    ) -> Callable[Concatenate[bool, P] R]:
        def wrapper(retry_once: bool, *args: P.args, **kwargs: P.kwargs) -> R:
            ...
        return wrapper

    ...


With our proposed syntax, the properly-typed decorator example becomes
concise and the type representations are visually descriptive::

    from typing import Any, ParamSpec, TypeVar

    R = TypeVar("R")
    P = ParamSpec("P")

    def with_retries(
        f: (**P) -> R
    ) -> (bool, **P) -> R:
        ...

Comparing to Other Languages
----------------------------

Many popular programming languages use an arrow syntax similar
to the one we are proposing here.

TypeScript
~~~~~~~~~~

In `TypeScript
<https://basarat.gitbook.io/typescript/type-system/callable#arrow-syntax>`_,
function types are expressed in a syntax almost the same as the one we
are proposing, but the arrow token is ``=>`` and arguments have names::

    (x: int, y: str) => bool

The names of the arguments are not actually relevant to the type. So,
for example, this is the same callable type::

    (a: int, b: str) => bool

Kotlin
~~~~~~

Function types in `Kotlin <https://kotlinlang.org/docs/lambdas.html>`_ permit
an identical syntax to the one we are proposing, for example::

    (Int, String) -> Bool

It also optionally allows adding names to the arguments, for example::

    (x: Int, y: String) -> Bool

As in TypeScript, the argument names (if provided) are just there for
documentation and are not part of the type itself.

Scala
~~~~~

`Scala <https://docs.scala-lang.org/tour/higher-order-functions.html>`_
uses the ``=>`` arrow for function types. Other than that, their syntax is
the same as the one we are proposing, for example::

    (Int, String) => Bool

Scala, like Python, has the ability to provide function arguments by name.
Function types can optionally include names, for example::

    (x: Int, y: String) => Bool

Unlike in TypeScript and Kotlin, these names are part of the type if
provided - any function implementing the type must use the same names.
This is similar to the extended syntax proposal we describe in our
`Rejected Alternatives`_ section.

Function Definitions vs Callable Type Annotations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In all of the languages listed above, type annotations for function
definitions use a ``:`` rather than a ``->``. For example, in TypeScript
a simple add function looks like this::

    function higher_order(fn: (a: string) => string): string {
      return fn("Hello, World");
    }

Scala and Kotlin use essentially the same ``:`` syntax for return
annotations.  The ``:`` makes sense in these languages because they
all use ``:`` for type annotations of
parameters and variables, and the use for function return types is
similar.

In Python we use ``:`` to denote the start of a function body and
``->`` for return annotations. As a result, even though our proposal
is superficially the same as these other languages the context is
different. There is potential for more confusion in Python when
reading function definitions that include callable types.

This is a key concern for which we are seeking feedback with our draft
PEP; one idea we have floated is to use ``=>`` instead to make it easier
to differentiate.


The ML Language Family
~~~~~~~~~~~~~~~~~~~~~~

Languages in the ML family, including `F#
<https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/fsharp-types#syntax-for-types>`_,
`OCaml
<https://www2.ocaml.org/learn/tutorials/basics.html#Defining-a-function>`_,
and `Haskell <https://wiki.haskell.org/Type_signature>`_, all use
``->`` to represent function types. All of them use a parentheses-free
syntax with multiple arrows, for example in Haskell::

    Integer -> String -> Bool

The use of multiple arrows, which differs from our proposal, makes
sense for languages in this family because they use automatic
`currying <https://en.wikipedia.org/wiki/Currying>`_ of function arguments,
which means that a multi-argument function behaves like a single-argument
function returning a function.

Specification
=============

Typing Behavior
---------------

Type checkers should treat the new syntax with exactly the same
semantics as ``typing.Callable``.

As such, a type checker should treat the following pairs exactly the
same::

    from typing import Awaitable, Callable, Concatenate, ParamSpec, TypeVarTuple

    P = ParamSpec("P")
    Ts = TypeVarTuple('Ts')

    f0: () -> bool
    f0: Callable[[], bool]

    f1: (int, str) -> bool
    f1: Callable[[int, str], bool]

    f2: (...) -> bool
    f2: Callable[..., bool]

    f3: async (str) -> str
    f3: Callable[[str], Awaitable[str]]

    f4: (**P) -> bool
    f4: Callable[P, bool]

    f5: (int, **P) -> bool
    f5: Callable[Concatenate[int, P], bool]

    f6: (*Ts) -> bool
    f6: Callable[[*Ts], bool]

    f7: (int, *Ts, str) -> bool
    f7: Callable[[int, *Ts, str], bool]


Grammar and AST
---------------

The proposed new syntax can be described by these AST changes to `Parser/Python.asdl
<https://github.com/python/cpython/blob/main/Parser/Python.asdl>`_::

    expr = <prexisting_expr_kinds>
         | AsyncCallableType(callable_type_arguments args, expr returns)
         | CallableType(callable_type_arguments args, expr returns)

    callable_type_arguments = AnyArguments
                            | ArgumentsList(expr* posonlyargs)
                            | Concatenation(expr* posonlyargs, expr param_spec)


Here are our proposed changes to the `Python Grammar
<https://docs.python.org/3/reference/grammar.htm>`::

    expression:
        | disjunction disjunction 'else' expression
        | callable_type_expression
        | disjunction
        | lambdef

    callable_type_expression:
        | callable_type_arguments '->' expression
        | ASYNC callable_type_arguments '->' expression

    callable_type_arguments:
        | '(' '...' [','] ')'
        | '(' callable_type_positional_argument*  ')'
        | '(' callable_type_positional_argument* callable_type_param_spec ')'

    callable_type_positional_argument:
        | !'...' expression ','
        | !'...' expression &')'

    callable_type_param_spec:
        | '**' expression ','
        | '**' expression &')'



If :pep:`646` is accepted, we intend to include support for unpacked
types in two ways. To support the "star-for-unpack" syntax proposed in
:pep:`646`, we will modify the grammar for
``callable_type_positional_argument`` as follows::

    callable_type_positional_argument:
        | !'...' expression ','
        | !'...' expression &')'
        | '*' expression ','
        | '*' expression &')'

With this change, a type of the form ``(int, *Ts) -> bool`` should
evaluate the AST form::

    CallableType(
        ArgumentsList(Name("int"), Starred(Name("Ts")),
        Name("bool")
    )

and be treated by type checkers as equivalent to or ``Callable[[int,
*Ts], bool]`` or ``Callable[[int, Unpack[Ts]], bool]``.


Implications of the Grammar
---------------------------

Precedence of ->
~~~~~~~~~~~~~~~~


``->`` binds less tightly than other operators, both inside types and
in function signatures, so the following two callable types are
equivalent::

    (int) -> str | bool
    (int) -> (str | bool)


``->`` associates to the right, both inside types and in function
signatures. So the following pairs are equivalent::

    (int) -> (str) -> bool
    (int) -> ((str) -> bool)

    def f() -> (int, str) -> bool: pass
    def f() -> ((int, str) -> bool): pass

    def f() -> (int) -> (str) -> bool: pass
    def f() -> ((int) -> ((str) -> bool)): pass


Because operators bind more tightly than ``->``, parentheses are
required whenever an arrow type is intended to be inside an argument
to an operator like ``|``::

    (int) -> () -> int | () -> bool      # syntax error!
    (int) -> (() -> int) | (() -> bool)  # okay


We discussed each of these behaviors and believe they are desirable:

- Union types (represented by ``A | B`` according to :pep:`604`) are
  valid in function signature returns, so we need to allow operators
  in the return position for consistency.
- Given that operators bind more tightly than ``->`` it is correct
  that a type like ``bool | () -> bool`` must be a syntax error. We
  should be sure the error message is clear because this may be a
  common mistake.
- Associating ``->`` to the right, rather than requiring explicit
  parentheses, is consistent with other languages like TypeScript and
  respects the principle that valid expressions should normally be
  substitutable when possible.

``async`` Keyword
~~~~~~~~~~~~~~~~~

All of the binding rules still work for async callable types::

    (int) -> async (float) -> str | bool
    (int) -> (async (float) -> (str | bool))

    def f() -> async (int, str) -> bool: pass
    def f() -> (async (int, str) -> bool): pass

    def f() -> async (int) -> async (str) -> bool: pass
    def f() -> (async (int) -> (async (str) -> bool)): pass


Trailing Commas
~~~~~~~~~~~~~~~

- Following the precedent of function signatures, putting a comma in
  an empty arguments list is illegal: ``(,) -> bool`` is a syntax
  error.
- Again following precedent, trailing commas are otherwise always
  permitted::


    ((int,) -> bool == (int) -> bool
    ((int, **P,) -> bool == (int, **P) -> bool
    ((...,) -> bool) == ((...) -> bool)

Allowing trailing commas also gives autoformatters more flexibility
when splitting callable types across lines, which is always legal
following standard python whitespace rules.


Disallowing ``...`` as an Argument Type
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Under normal circumstances, any valid expression is permitted where we
want a type annotation and ``...`` is a valid expression. This is
never semantically valid and all type checkers would reject it, but
the grammar would allow it if we did not explicitly prevent this.

Since ``...`` is meaningless as a type and there are usability
concerns, our grammar rules it out and the following is a syntax
error::

    (int, ...) -> bool

We decided that there were compelling reasons to do this:

- The semantics of ``(...) -> bool`` are different from ``(T) -> bool``
  for any valid type T: ``(...)`` is a special form indicating
  ``AnyArguments`` whereas ``T`` is a type parameter in the arguments
  list.
- ``...`` is used as a placeholder default value to indicate an
  optional argument in stubs and callback protocols. Allowing it in
  the position of a type could easily lead to confusion and possibly
  bugs due to typos.
- In the ``tuple`` generic type, we special-case ``...`` to mean
  "more of the same", e.g. a ``tuple[int, ...]`` means a tuple with
  one or more integers. We do not use ``...`` in a a similar way
  in callable types, so to prevent misunderstandings it makes sense
  to prevent this.



Incompatibility with other possible uses of ``*`` and ``**``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The use of ``**P`` for supporting :pep:`612` ``ParamSpec`` rules out any
future proposal using a bare ``**<some_type>`` to type
``kwargs``. This seems acceptable because:

- If we ever do want such a syntax, it would be clearer to require an
  argument name anyway. This would also make the type look more
  similar to a function signature. In other words, if we ever support
  typing ``kwargs`` in callable types, we would prefer ``(int,
  **kwargs: str)`` rather than ``(int, **str)``.
- :pep:`646` unpacking syntax would rule out using ``*<some_type>`` for
  ``args``. The ``kwargs`` case is similar enough that this rules out
  a bare ``**<some_type>`` anyway.



Compatibility with Arrow-Based Lambda Syntax
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To the best of our knowledge there is no active discussion of
arrow-style lambda syntax that we are aware of, but it is nonetheless
worth considering what possibilities would be ruled out by adopting
this proposal.

It would be incompatible with this proposal to adopt the same a
parenthesized ``->``-based arrow syntax for lambdas, e.g.  ``(x, y) ->
x + y`` for ``lambda x, y: x + y``.


Our view is that if we want arrow syntax for lambdas in the future, it
would be a better choice to use ``=>``, e.g. ``(x, y) => x + y``.
Many languages use the same arrow token for both lambdas and callable
types, but Python is unique in that types are expressions and have to
evaluate to runtime values. Our view is that this merits using
separate tokens, and given the existing use of ``->`` for return types
in function signatures it would be more coherent to use ``->`` for
callable types and ``=>`` for lambdas.

Runtime Behavior
----------------

The new AST nodes need to evaluate to runtime types, and we have two goals for the
behavior of these runtime types:

- They should expose a structured API that is descriptive and powerful
  enough to be compatible with extending the type to include new features
  like named and variadic arguments.
- They should also expose an API that is backward-compatible with
  ``typing.Callable``.

Evaluation and Structured API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We intend to create new builtin types to which the new AST nodes will
evaluate, exposing them in the ``types`` module.

Our plan is to expose a structured API as if they were defined as follows::

    class CallableType:
        is_async: bool
        arguments: Ellipsis | tuple[CallableTypeArgument]
        return_type: object

    class CallableTypeArgument:
        kind: CallableTypeArgumentKind
        annotation: object

    @enum.global_enum
    class CallableTypeArgumentKind(enum.IntEnum):
        POSITIONAL_ONLY: int = ...
        PARAM_SPEC: int = ...


The evaluation rules are expressed in terms of the following
pseudocode::

    def evaluate_callable_type(
        callable_type: ast.CallableType | ast.AsyncCallableType:
    ) -> CallableType:
        return CallableType(
           is_async=isinstance(callable_type, ast.AsyncCallableType),
           arguments=_evaluate_arguments(callable_type.arguments),
           return_type=evaluate_expression(callable_type.returns),
        )

    def _evaluate_arguments(arguments):
        match arguments:
            case ast.AnyArguments():
                return Ellipsis
            case ast.ArgumentsList(posonlyargs):
                return tuple(
                    _evaluate_arg(arg) for arg in args
                )
            case ast.ArgumentsListConcatenation(posonlyargs, param_spec):
                return tuple(
                    *(evaluate_arg(arg) for arg in args),
                    _evaluate_arg(arg=param_spec, kind=PARAM_SPEC)
                )
            if isinstance(arguments, Any
        return Ellipsis

    def _evaluate_arg(arg, kind=POSITIONAL_ONLY):
        return CallableTypeArgument(
            kind=POSITIONAL_ONLY,
            annotation=evaluate_expression(value)
        )


Backward-Compatible API
~~~~~~~~~~~~~~~~~~~~~~~

To get backward compatibility with the existing ``types.Callable`` API,
which relies on fields ``__args__`` and ``__parameters__``, we can define
them as if they were written in terms of the following::

    import itertools
    import typing

    def get_args(t: CallableType) -> tuple[object]:
        return_type_arg = (
            typing.Awaitable[t.return_type]
            if t.is_async
            else t.return_type
        )
        arguments = t.arguments
        if isinstance(arguments, Ellipsis):
            argument_args = (Ellipsis,)
        else:
            argument_args = (arg.annotation for arg in arguments)
        return (
            *arguments_args,
            return_type_arg
        )

    def get_parameters(t: CallableType) -> tuple[object]:
        out = []
        for arg in get_args(t):
            if isinstance(arg, typing.ParamSpec):
                out.append(t)
            else:
                out.extend(arg.__parameters__)
        return tuple(out)


Additional Behaviors of ``types.CallableType``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

As with the ``A | B`` syntax for unions introduced in :pep:`604`:

- The ``__eq__`` method should treat equivalent ``typing.Callable``
  values as equal to values constructed using the builtin syntax, and
  otherwise should behave like the ``__eq__`` of ``typing.Callable``.
- The ``__repr__`` method should produce an arrow syntax representation that,
  when evaluated, gives us back an equal ``types.CallableType`` instance.


Rejected Alternatives
=====================

Many of the alternatives we considered would have been more expressive
than ``typing.Callable``, for example adding support for describing
signatures that include named, optional, and variadic arguments.

To determine which features we most needed to support with a callable
type syntax, we did an extensive analysis of existing projects:

- `stats on the use of the Callable type <https://github.com/pradeep90/annotation_collector#typed-projects---callable-type>`_;
- `stats on how untyped and partially-typed callbacks are actually used <https://github.com/pradeep90/annotation_collector#typed-projects---callback-usage>`_.

We decided on a simple proposal with improved syntax for the existing
``Callable`` type because the vast majority of callbacks can be correctly
described by the existing ``typing.Callable`` semantics:

- Positional parameters: By far the most important case to handle well
  is simple callable types with positional parameters, such as
  ``(int, str) -> bool``
- ParamSpec and Concatenate: The next most important feature is good
  support for :pep:`612` ``ParamSpec`` and ``Concatenate`` types like
  ``(**P) -> bool`` and ``(int, **P) -> bool``. These are common
  primarily because of the heavy use of decorator patterns in python
  code.
- TypeVarTuples: The next most important feature, assuming :pep:`646` is
  accepted, is for unpacked types which are common because of cases
  where a wrapper passes along ``*args`` to some other function.

Features that other, more complicated proposals would support account
for fewer than 2% of the use cases we found. These are already
expressible using callback protocols, and since they are uncommon we
decided that it made more sense to move forward with a simpler syntax.

Extended Syntax Supporting Named and Optional Arguments
-------------------------------------------------------

Another alternative was for a compatible but more complex syntax that
could express everything in this PEP but also named, optional, and
variadic arguments. In this “extended” syntax proposal the following
types would have been equivalent::

    class Function(typing.Protocol):
        def f(self, x: int, /, y: float, *, z: bool = ..., **kwargs: str) -> bool:
            ...

    Function = (int, y: float, *, z: bool = ..., **kwargs: str) -> bool

Advantages of this syntax include: - Most of the advantages of the
proposal in this PEP (conciseness, :pep:`612` support, etc) -
Furthermore, the ability to handle named, optional, and variadic
arguments

We decided against proposing it for the following reasons:

- The implementation would have been more difficult, and usage stats
  demonstrate that fewer than 3% of use cases would benefit from any
  of the added features.
- The group that debated these proposals was split down the middle
  about whether these changes are desirable:

  - On the one hand, they make callable types more expressive. On the
    other hand, they could easily confuse users who have not read the
    full specification of callable type syntax.
  - We believe the simpler syntax proposed in this PEP, which
    introduces no new semantics and closely mimics syntax in other
    popular languages like Kotlin, Scala, and TypesScript, is much
    less likely to confuse users.

- We intend to implement the current proposal in a way that is
  forward-compatible with the more complicated extended syntax. If the
  community decides after more experience and discussion that we want
  the additional features, it should be straightforward to propose
  them in the future.
- Even a full extended syntax cannot replace the use of callback
  protocols for overloads. For example, no closed form of callable type
  could express a function that maps bools to bools and ints to floats,
  like this callback protocol.::

    from typing import overload, Protocol

    class OverloadedCallback(Protocol)

      @overload
      def __call__(self, x: int) -> float: ...

      @overload
      def __call__(self, x: bool) -> bool: ...

      def __call__(self, x: int | bool) -> float | bool: ...


    f: OverloadedCallback = ...
    f(True)  # bool
    f(3)     # float



We confirmed that the current proposal is forward-compatible with
extended syntax by
`implementing <https://github.com/stroxler/cpython/tree/callable-type-syntax--extended>`_
a grammar and AST for this extended syntax on top of our reference
implementation of this PEP's grammar.


Syntax Closer to Function Signatures
------------------------------------

One alternative we had floated was a syntax much more similar to
function signatures.

In this proposal, the following types would have been equivalent::

    class Function(typing.Protocol):
        def f(self, x: int, /, y: float, *, z: bool = ..., **kwargs: str) -> bool:
            ...

    Function = (x: int, /, y: float, *, z: bool = ..., **kwargs: str) -> bool


The benefits of this proposal would have included:

- Perfect syntactic consistency between signatures and callable types.
- Support for more features of function signatures (named, optional,
  variadic args) that this PEP does not support.

Key downsides that led us to reject the idea include the following:

- A large majority of use cases only use positional-only arguments.  This
  syntax would be more verbose for that use case, both because of requiring
  argument names and an explicit ``/``, for example ``(int, /) -> bool`` where
  our proposal allows ``(int) -> bool``
- The requirement for explicit ``/`` for positional-only arguments has
  a high risk of causing frequent bugs - which often would not be
  detected by unit tests - where library authors would accidentally
  use types with named arguments.
- Our analysis suggests that support for ``ParamSpec`` is key, but the
  scoping rules laid out in :pep:`612` would have made this difficult.


Other Proposals Considered
--------------------------

Functions-as-Types
~~~~~~~~~~~~~~~~~~

An idea we looked at very early on was to `allow using functions as types
<https://docs.google.com/document/d/1rv6CCDnmLIeDrYlXe-QcyT0xNPSYAuO1EBYjU3imU5s/edit?usp=sharing>`_.
The idea is allowing a function to stand in for its own call
signature, with roughly the same semantics as the ``__call__`` method
of callback protocols::

    def CallableType(
        positional_only: int,
        /,
        named: str,
        *args: float,
        keyword_only: int = ...,
        **kwargs: str
    ) -> bool: ...

    f: CallableType = ...
    f(5, 6.6, 6.7, named=6, x="hello", y="world")  # typechecks as bool

This may be a good idea, but we do not consider it a viable
replacement for callable types:

- It would be difficult to handle ``ParamSpec``, which we consider a
  critical feature to support.
- When using functions as types, the callable types are not first-class
  values.  Instead, they require a separate, out-of-line function
  definition to define a type alias
- It would not support more features than callback protocols, and seems
  more like a shorter way to write them than a replacement for
  ``Callable``.

Hybrid keyword-arrow Syntax
~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the Rust language, a keyword ``fn`` is used to indicate functions
in much the same way as Python's ``def``, and callable types are
indicated using a hybrid arrow syntax ``Fn(i64, String) -> bool``.

We could use the ``def`` keyword in callable types for Python, for
example our two-parameter boolean function could be written as
``def(int, str) -> bool``. But we think this might confuse readers
into thinking ``def(A, B) -> C`` is a lambda, particularly because
Javascript's ``function`` keyword is used in both named and anonymous
functions.

Parenthesis-Free Syntax
~~~~~~~~~~~~~~~~~~~~~~~

We considered a parentheses-free syntax that would have been even more
concise::

    int, str -> bool

We decided against it because this is not visually as similar to
existing function header syntax. Moreover, it is visually similar to
lambdas, which bind names with no parentheses: ``lambda x, y: x ==
y``.

Requiring Outer Parentheses
~~~~~~~~~~~~~~~~~~~~~~~~~~~

A concern with the current proposal is readability, particularly
when callable types are used in return type position which leads to
multiple top-level ``->`` tokens, for example::

    def make_adder() -> (int) -> int:
        return lambda x: x + 1

We considered a few ideas to prevent this by changing rules about
parentheses. One was to move the parentheses to the outside, so
that a two-argument boolean function is written ``(int, str -> bool)``.
With this change, the example above becomes::

    def make_adder() -> (int -> int):
        return lambda x: x + 1

This makes the nesting of many examples that are difficult to
follow clear, but we rejected it because

- Currently in Python commas bind very loosely, which means it might be common
  to misread ``(int, str -> bool)`` as a tuple whose first element is an int,
  rather than a two-parameter callable type.
- It is not very similar to function header syntax, and one of our goals was
  familiar syntax inspired by function headers.
- This syntax may be more readable for deaply nested callables like the one
  above, but deep nesting is not very common. Encouraging extra parentheses
  around callable types in return position via a style guide would have most of
  the readability benefit without the downsides.

We also considered requiring parentheses on both the parameter list and the
outside, e.g. ``((int, str) -> bool)``. With this change, the example above
becomes::

    def make_adder() -> ((int) -> int):
        return lambda x: x + 1

We rejected this change because:

- The outer parentheses only help readability in some cases, mostly when a
  callable type is used in return position. In many other cases they hurt
  readability rather than helping.
- We agree that it might make sense to encourage outer parentheses in several
  cases, particularly callable types in function return annotations. But

  - We believe it is more appropriate to encourage this in style guides,
    linters, and autoformatters than to bake it into the parser and throw
    syntax errors.
  - Moreover, if a type is complicated enough that readability is a concern
    we can always use type aliases, for example::

        IntToIntFunction: (int) -> int

        def make_adder() -> IntToIntFunction:
            return lambda x: x + 1


Making ``->`` bind tighter than ``|``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In order to allow both ``->`` and ``|`` tokens in type expressions we
had to choose precedence. In the current proposal, this is a function
returning an optional boolean::

    (int, str) -> bool | None  # equivalent to (int, str) -> (bool | None)

We considered having ``->`` bind tighter so that instead the expression
would parse as ``((int, str) -> bool) | None``. There are two advantages
to this:

- It means we no would longer have to treat ``None | (int, str) ->
  bool`` as a syntax error.
- Looking at typeshed today, optional callable arguments are very common
  because using ``None`` as a default value is a standard Python idiom.
  Having ``->`` bind tighter would make these easier to write.

We decided against this for a few reasons:

- The function header ``def f() -> int | None: ...`` is legal
  and indicates a function returning an optional int. To be consistent
  with function headers, callable types should do the same.
- TypeScript is the other popular language we know of that uses both
  ``->`` and ``|`` tokens in type expressions, and they have ``|`` bind
  tighter. While we do not have to follow their lead, we prefer to do
  so.
- We do acknowledge that optional callable types are common and
  having ``|`` bind tighter forces extra parentheses, which makes these
  types harder to write. But code is read more often than written, and
  we believe that requiring the outer parentheses for an optional callable
  type like ``((int, str) -> bool) | None`` is preferable for readability.


Introducing type-strings
~~~~~~~~~~~~~~~~~~~~~~~~

Another idea was adding a new “special string” syntax and putting the type
inside of it, for example ``t”(int, str) -> bool”``. We rejected this
because it is not as readable, and seems out of step with `guidance
<https://mail.python.org/archives/list/python-dev@python.org/message/SZLWVYV2HPLU6AH7DOUD7DWFUGBJGQAY/>`_
from the Steering Council on ensuring that type expressions do not
diverge from the rest of Python's syntax.


Improving Usability of the Indexed Callable Type
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If we do not want to add new syntax for callable types, we could
look at how to make the existing type easier to read. One proposal
would be to make the builtin ``callable`` function indexable so
that it could be used as a type::

    callable[[int, str], bool]

This change would be analogous to :pep:`585` that made built in collections
like ``list`` and ``dict`` usable as types, and would make imports
more convenient, but it wouldn't help readability of the types themselves
much.

In order to reduce the number of brackets needed in complex callable
types, it would be possible to allow tuples for the argument list::

    callable[(int, str), bool]

This actually is a significant readability improvement for
multi-argument functions, but the problem is that it makes callables
with one arguments, which are the most common arity, hard to
write: because ``(x)`` evaluates to ``x``, they would have to be
written like ``callable[(int,), bool]``. We find this awkward.

Moreover, none of these ideas help as much with reducing verbosity
as the current proposal, nor do they introduce as strong a visual cue
as the ``->`` between the parameter types and the return type.

Alternative Runtime Behaviors
-----------------------------

The hard requirements on our runtime API are that:

- It must preserve backward compatibility with ``typing.Callable`` via
  ``__args__`` and ``__params__``.
- It must provide a structured API, which should be extensible if
  in the future we try to support named and variadic arguments.

Alternative APIs
~~~~~~~~~~~~~~~~

We considered having the runtime data ``types.CallableType`` use a
more structured API where there would be separate fields for
``posonlyargs`` and ``param_spec``. The current proposal was
was inspired by the ``inspect.Signature`` type.

We use "argument" in our field and type names, unlike "parameter"
as in ``inspect.Signature``, in order to avoid confusion with
the ``callable_type.__parameters__`` field from the legacy API
that refers to type parameters rather than callable parameters.

Using the plain return type in ``__args__`` for async types
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

It is debatable whether we are required to preserve backward compatibility
of ``__args__`` for async callable types like ``async (int) -> str``. The
reason is that one could argue they are not expressible directly
using ``typing.Callable``, and therefore it would be fine to set
``__args__`` as ``(int, int)`` rather than ``(int, typing.Awaitable[int])``.

But we believe this would be problematic. By preserving the appearance
of a backward-compatible API while actually breaking its semantics on
async types, we would cause runtime type libraries that attempt to
interpret ``Callable`` using ``__args__`` to fail silently.

It is for this reason that we automatically wrap the return type in
``Awaitable``.

Backward Compatibility
======================

This PEP proposes a major syntax improvement over ``typing.Callable``,
but the static semantics are the same.

As such, the only thing we need for backward compatibility is to
ensure that types specified via the new syntax behave the same as
equivalent ``typing.Callable`` and ``typing.Concatenate`` values they
intend to replace.

There is no particular interaction between this proposal and ``from
__future__ import annotations`` - just like any other type annotation
it will be unparsed to a string at module import, and
``typing.get_type_hints`` should correctly evaluate the resulting
strings in cases where that is possible.

This is discussed in more detail in the Runtime Behavior section.


Reference Implementation
========================

We have a working `implementation
<https://github.com/stroxler/cpython/tree/callable-type-syntax--shorthand>`_
of the AST and Grammar with tests verifying that the grammar proposed
here has the desired behaviors.

The runtime behavior is not yet implemented. As discussed in the
`Runtime Behavior`_ portion of the spec we have a detailed plan for
both a backward-compatible API and a more structured API in
`a separate doc
<https://docs.google.com/document/d/15nmTDA_39Lo-EULQQwdwYx_Q1IYX4dD5WPnHbFG71Lk/edit>`_
where we are also open to discussion and alternative ideas.


Open Issues
===========

Details of the Runtime API
--------------------------

We have attempted to provide a complete behavior specification in
the `Runtime Behavior`_ section of this PEP.

But there are probably more details that we will not realize we
need to define until we build a full reference implementation.

Optimizing ``SyntaxError`` messages
-----------------------------------

The current reference implementation has a fully-functional parser and
all edge cases presented here have been tested.

But there are some known cases where the errors are not as informative
as we would like. For example, because ``(int, ...) -> bool`` is
illegal but ``(int, ...)`` is a valid tuple, we currently produce a
syntax error flagging the ``->`` as the problem even though the real
cause of the error is using ``...`` as an argument type.

This is not part of the specification *per se* but is an important
detail to address in our implementation. The solution will likely
involve adding ``invalid_.*`` rules to ``python.gram`` and customizing
error messages.

Resources
=========

Background and History
----------------------

:pep:`PEP 484 specifies
<484#suggested-syntax-for-python-2-7-and-straddling-code>`
a very similar syntax for function type hint *comments* for use in
code that needs to work on Python 2.7. For example::

    def f(x, y):
        # type: (int, str) -> bool
        ...

At that time we used indexing operations to specify generic types like
``typing.Callable`` because we decided not to add syntax for
types. However, we have since begun to do so, e.g. with :pep:`604`.

**Maggie** proposed better callable type syntax as part of a larger
`presentation on typing simplifications
<https://drive.google.com/file/d/1XhqTKoO6RHtz7zXqW5Wgq9nzaEz9TXjI/view>`_
at the PyCon Typing Summit 2021.

**Steven** `brought up this proposal on typing-sig
<https://mail.python.org/archives/list/typing-sig@python.org/thread/3JNXLYH5VFPBNIVKT6FFBVVFCZO4GFR2>`_.
We had several meetings to discuss alternatives, and `this presentation
<https://www.dropbox.com/s/sshgtr4p30cs0vc/Python%20Callable%20Syntax%20Proposals.pdf?dl=0>`_
led us to the current proposal.

**Pradeep** `brought this proposal to python-dev
<https://mail.python.org/archives/list/python-dev@python.org/thread/VBHJOS3LOXGVU6I4FABM6DKHH65GGCUB>`_
for feedback.

Acknowledgments
---------------

Thanks to the following people for their feedback on the PEP and help
planning the reference implementation:

Alex Waygood, Eric Traut, Guido van Rossum, James Hilton-Balfe,
Jelle Zijlstra, Maggie Moss, Tuomas Suutari, Shannon Zhu.

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: