From 4a3a9932cb991623846e44603a2427313b04ebe1 Mon Sep 17 00:00:00 2001 From: Steven Troxler Date: Tue, 18 Jan 2022 10:34:53 -0800 Subject: [PATCH] PEP 677: Runtime Behavior Specification (#2237) Co-authored-by: Jelle Zijlstra --- pep-0677.rst | 179 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 158 insertions(+), 21 deletions(-) diff --git a/pep-0677.rst b/pep-0677.rst index 41fe57b92..79a2b09b4 100644 --- a/pep-0677.rst +++ b/pep-0677.rst @@ -635,24 +635,120 @@ callable types and ``=>`` for lambdas. Runtime Behavior ---------------- -Our tentative plan is that: +The new AST nodes need to evaluate to runtime types, and we have two goals for the +behavior of these runtime types: -- The ``__repr__`` will show an arrow syntax literal. -- We will provide a new API where the runtime data structure can be - accessed in the same manner as the AST data structure. -- We will ensure that we provide an API that is backward-compatible - with ``typing.Callable`` and ``typing.Concatenate``, specifically - the behavior of ``__args__`` and ``__parameters__``. +- They should expose a structured API that is descriptive and powerful + enough to be compatible with extending the type to include new features + like named and variadic arguments. +- They should also expose an API that is backward-compatible with + ``typing.Callable``. -Because these details are still under debate we are currently -maintaining `a separate doc -`_ -with details about the new builtins, the evaluation model, how to -provide both a backward-compatible and more structured API, and -possible alternatives to the current plan. +Evaluation and Structured API +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We intend to create new builtin types to which the new AST nodes will +evaluate, exposing them in the ``types`` module. + +Our plan is to expose a structured API as if they were defined as follows:: + + class CallableType: + is_async: bool + arguments: Ellipsis | tuple[CallableTypeArgument] + return_type: object + + class CallableTypeArgument: + kind: CallableTypeArgumentKind + annotation: object + + @enum.global_enum + class CallableTypeArgumentKind(enum.IntEnum): + POSITIONAL_ONLY: int = ... + PARAM_SPEC: int = ... + + +The evaluation rules are expressed in terms of the following +pseudocode:: + + def evaluate_callable_type( + callable_type: ast.CallableType | ast.AsyncCallableType: + ) -> CallableType: + return CallableType( + is_async=isinstance(callable_type, ast.AsyncCallableType), + arguments=_evaluate_arguments(callable_type.arguments), + return_type=evaluate_expression(callable_type.returns), + ) + + def _evaluate_arguments(arguments): + match arguments: + case ast.AnyArguments(): + return Ellipsis + case ast.ArgumentsList(posonlyargs): + return tuple( + _evaluate_arg(arg) for arg in args + ) + case ast.ArgumentsListConcatenation(posonlyargs, param_spec): + return tuple( + *(evaluate_arg(arg) for arg in args), + _evaluate_arg(arg=param_spec, kind=PARAM_SPEC) + ) + if isinstance(arguments, Any + return Ellipsis + + def _evaluate_arg(arg, kind=POSITIONAL_ONLY): + return CallableTypeArgument( + kind=POSITIONAL_ONLY, + annotation=evaluate_expression(value) + ) + + +Backward-Compatible API +~~~~~~~~~~~~~~~~~~~~~~~ + +To get backward compatibility with the existing ``types.Callable`` API, +which relies on fields ``__args__`` and ``__parameters__``, we can define +them as if they were written in terms of the following:: + + import itertools + import typing + + def get_args(t: CallableType) -> tuple[object]: + return_type_arg = ( + typing.Awaitable[t.return_type] + if t.is_async + else t.return_type + ) + arguments = t.arguments + if isinstance(arguments, Ellipsis): + argument_args = (Ellipsis,) + else: + argument_args = (arg.annotation for arg in arguments) + return ( + *arguments_args, + return_type_arg + ) + + def get_parameters(t: CallableType) -> tuple[object]: + out = [] + for arg in get_args(t): + if isinstance(arg, typing.ParamSpec): + out.append(t) + else: + out.extend(arg.__parameters__) + return tuple(out) + + +Additional Behaviors of ``types.CallableType`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As with the ``A | B`` syntax for unions introduced in PEP 604: + +- The ``__eq__`` method should treat equivalent ``typing.Callable`` + values as equal to values constructed using the builtin syntax, and + otherwise should behave like the ``__eq__`` of ``typing.Callable``. +- The ``__repr__`` method should produce an arrow syntax representation that, + when evaluated, gives us back an equal ``types.CallableType`` instance. -Once the plan is finalized we will include a full specification of -runtime behavior in this section of the PEP. Rejected Alternatives ===================== @@ -908,9 +1004,9 @@ We rejected this change because: syntax errors. - Moreover, if a type is complicated enough that readability is a concern we can always use type aliases, for example:: - + IntToIntFunction: (int) -> int - + def make_adder() -> IntToIntFunction: return lambda x: x + 1 @@ -991,6 +1087,46 @@ Moreover, none of these ideas help as much with reducing verbosity as the current proposal, nor do they introduce as strong a visual cue as the ``->`` between the parameter types and the return type. +Alternative Runtime Behaviors +----------------------------- + +The hard requirements on our runtime API are that: + +- It must preserve backward compatibility with ``typing.Callable`` via + ``__args__`` and ``__params__``. +- It must provide a structured API, which should be extensible if + in the future we try to support named and variadic arguments. + +Alternative APIs +~~~~~~~~~~~~~~~~ + +We considered having the runtime data ``types.CallableType`` use a +more structured API where there would be separate fields for +``posonlyargs`` and ``param_spec``. The current proposal was +was inspired by the ``inspect.Signature`` type. + +We use "argument" in our field and type names, unlike "parameter" +as in ``inspect.Signature``, in order to avoid confusion with +the ``callable_type.__parameters__`` field from the legacy API +that refers to type parameters rather than callable parameters. + +Using the plain return type in ``__args__`` for async types +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It is debatable whether we are required to preserve backward compatiblity +of ``__args__`` for async callable types like ``async (int) -> str``. The +reason is that one could argue they are not expressible directly +using ``typing.Callable``, and therefore it would be fine to set +``__args__`` as ``(int, int)`` rather than ``(int, typing.Awaitable[int])``. + +But we believe this would be problematic. By preserving the appearance +of a backward-compatible API while actually breaking its semantics on +async types, we would cause runtime type libraries that attempt to +interpret ``Callable`` using ``__args__`` to fail silently. + +It is for this reason that we automatically wrap the return type in +``Awaitable``. + Backward Compatibility ====================== @@ -1033,10 +1169,11 @@ Open Issues Details of the Runtime API -------------------------- -Once we have finalized all details of the runtime behavior, we -will need to add a full specification of the behavior to the -`Runtime Behavior`_ section of this PEP as well as include that -behavior in our reference implementation. +We have attempted to provide a complete behavior specification in +the `Runtime Behavior`_ section of this PEP. + +But there are probably more details that we will not realize we +need to define until we build a full reference implementation. Optimizing ``SyntaxError`` messages -----------------------------------