PEP: 696 Title: Type defaults for TypeVarLikes Author: James Hilton-Balfe Sponsor: Jelle Zijlstra Discussions-To: https://mail.python.org/archives/list/typing-sig@python.org/thread/7VWBZWXTCX6RAJO6GG67BAXUPFZ24NTC Status: Draft Type: Standards Track Topic: Typing Content-Type: text/x-rst Created: 14-Jul-2022 Python-Version: 3.12 Post-History: `22-Mar-2022 `__, `08-Jan-2023 `__, Abstract -------- This PEP introduces the concept of type defaults for ``TypeVarLike``\ s (``TypeVar``, ``ParamSpec`` and ``TypeVarTuple``), which act as defaults for a type parameter when one is not specified or the constraint solver isn't able to solve a type parameter to anything. Default type argument support is available in some popular languages such as C++, TypeScript, and Rust. A survey of type parameter syntax in some common languages has been conducted by the author of :pep:`695` and can be found in its :pep:`Appendix A <695#appendix-a-survey-of-type-parameter-syntax>`. Motivation ---------- .. code-block:: py T = TypeVar("T", default=int) # This means that if no type is specified T = int @dataclass class Box(Generic[T]): value: T | None = None reveal_type(Box()) # type is Box[int] reveal_type(Box(value="Hello World!")) # type is Box[str] One place this `regularly comes up `__ is ``Generator``. I propose changing the *stub definition* to something like: .. code-block:: py YieldT = TypeVar("YieldT") SendT = TypeVar("SendT", default=None) ReturnT = TypeVar("ReturnT", default=None) class Generator(Generic[YieldT, SendT, ReturnT]): ... Generator[int] == Generator[int, None] == Generator[int, None, None] This is also useful for a ``Generic`` that is commonly over one type. .. code-block:: py class Bot: ... BotT = TypeVar("BotT", bound=Bot, default=Bot) class Context(Generic[BotT]): bot: BotT class MyBot(Bot): ... reveal_type(Context().bot) # type is Bot # notice this is not Any which is what it would be currently reveal_type(Context[MyBot]().bot) # type is MyBot Not only does this improve typing for those who explicitly use it, it also helps non-typing users who rely on auto-complete to speed up their development. This design pattern is common in projects like: - `discord.py `__ — where the example above was taken from. - `NumPy `__ — the default for types like ``ndarray``'s ``dtype`` would be ``float64``. Currently it's ``Unknown`` or ``Any``. - `TensorFlow `__ — this could be used for Tensor similarly to ``numpy.ndarray`` and would be useful to simplify the definition of ``Layer``. Specification ------------- Default Ordering and Subscription Rules ''''''''''''''''''''''''''''''''''''''' The order for defaults should follow the standard function parameter rules, so a ``TypeVarLike`` with no ``default`` cannot follow one with a ``default`` value. Doing so should ideally raise a ``TypeError`` in ``typing._GenericAlias``/``types.GenericAlias``, and a type checker should flag this an error. .. code-block:: py DefaultStrT = TypeVar("DefaultStrT", default=str) DefaultIntT = TypeVar("DefaultIntT", default=int) DefaultBoolT = TypeVar("DefaultBoolT", default=bool) T = TypeVar("T") T2 = TypeVar("T2") class NonDefaultFollowsDefault(Generic[DefaultStrT, T]): ... # Invalid: non-default TypeVars cannot follow ones with defaults class NoNonDefaults(Generic[DefaultStrT, DefaultIntT]): ... ( NoNoneDefaults == NoNoneDefaults[str] == NoNoneDefaults[str, int] ) # All valid class OneDefault(Generic[T, DefaultBoolT]): ... OneDefault[float] == OneDefault[float, bool] # Valid reveal_type(OneDefault) # type is type[OneDefault[T, DefaultBoolT = bool]] reveal_type(OneDefault[float]()) # type is OneDefault[float, bool] class AllTheDefaults(Generic[T1, T2, DefaultStrT, DefaultIntT, DefaultBoolT]): ... reveal_type(AllTheDefaults) # type is type[AllTheDefaults[T1, T2, DefaultStrT = str, DefaultIntT = int, DefaultBoolT = bool]] reveal_type(AllTheDefaults[int, complex]()) # type is AllTheDefaults[int, complex, str, int, bool] AllTheDefaults[int] # Invalid: expected 2 arguments to AllTheDefaults ( AllTheDefaults[int, complex] == AllTheDefaults[int, complex, str] == AllTheDefaults[int, complex, str, int] == AllTheDefaults[int, complex, str, int, bool] ) # All valid This cannot be enforced at runtime for functions, for now, but in the future, this might be possible (see `Interaction with PEP 695 <#interaction-with-pep-695>`__). ``ParamSpec`` Defaults '''''''''''''''''''''' ``ParamSpec`` defaults are defined using the same syntax as ``TypeVar`` \ s but use a ``list`` of types or an ellipsis literal "``...``" or another in-scope ``ParamSpec`` (see :ref:`696-scoping-rules`). .. code-block:: py DefaultP = ParamSpec("DefaultP", default=[str, int]) class Foo(Generic[DefaultP]): ... reveal_type(Foo) # type is type[Foo[DefaultP = [str, int]]] reveal_type(Foo()) # type is Foo[[str, int]] reveal_type(Foo[[bool, bool]]()) # type is Foo[[bool, bool]] ``TypeVarTuple`` Defaults ''''''''''''''''''''''''' ``TypeVarTuple`` defaults are defined using the same syntax as ``TypeVar`` \ s but use an unpacked tuple of types instead of a single type or another in-scope ``TypeVarTuple`` (see :ref:`696-scoping-rules`). .. code-block:: py DefaultTs = TypeVarTuple("DefaultTs", default=Unpack[tuple[str, int]]) class Foo(Generic[*DefaultTs]): ... reveal_type(Foo) # type is type[Foo[DefaultTs = *tuple[str, int]]] reveal_type(Foo()) # type is Foo[str, int] reveal_type(Foo[int, bool]()) # type is Foo[int, bool] Using Another ``TypeVarLike`` as ``default`` '''''''''''''''''''''''''''''''''''''''''''' This allows for a value to be used again when the constraints solver fails to solve a constraint for a type, or the type parameter to a generic is missing but another type parameter is specified. To use another ``TypeVarLike`` as a default the ``default`` and the ``TypeVarLike`` must be the same type (a ``TypeVar``'s default must be a ``TypeVar``, etc.). `This could be used on builtins.slice `__ where the ``start`` parameter should default to ``int``, ``stop`` default to the type of ``start`` and step default to ``int | None``. .. code-block:: py StartT = TypeVar("StartT", default=int) StopT = TypeVar("StopT", default=StartT) StepT = TypeVar("StepT", default=int | None) class slice(Generic[StartT, StopT, StepT]): ... reveal_type(slice) # type is type[slice[StartT = int, StopT = StartT, StepT = int | None]] reveal_type(slice()) # type is slice[int, int, int | None] reveal_type(slice[str]()) # type is slice[str, str, int | None] reveal_type(slice[str, bool, timedelta]()) # type is slice[str, bool, timedelta] T2 = TypeVar("T2", default=DefaultStrT) class Foo(Generic[DefaultStrT, T2]): def __init__(self, a: DefaultStrT, b: T2) -> None: ... reveal_type(Foo(1, "")) # type is Foo[int, str] Foo[int](1, "") # Invalid: Foo[int, str] cannot be assigned to self: Foo[int, int] in Foo.__init__ Foo[int]("", 1) # Invalid: Foo[str, int] cannot be assigned to self: Foo[int, int] in Foo.__init__ When using a ``TypeVarLike`` as the default to another ``TypeVarLike``. Where ``T1`` is the default for ``T2`` the following rules apply. ``TypeVarTuple``\s are not supported because: - :ref:`696-scoping-rules` does not allow usage of ``TypeVarLikes`` from outer scopes. - Multiple ``TypeVarTuple``\s cannot appear in the type parameter list for a single class, as specified in :pep:`646#multiple-type-variable-tuples-not-allowed`. - ``TypeVarLike`` defaults in functions are not supported. These reasons leave no current valid location where a ``TypeVarTuple`` could have a default. .. _696-scoping-rules: Scoping Rules ~~~~~~~~~~~~~ ``T1`` must be used before ``T2`` in the parameter list of the generic. .. code-block:: py DefaultT = TypeVar("DefaultT", default=T) class Foo(Generic[T, DefaultT]): ... # Valid class Foo(Generic[T]): class Bar(Generic[DefaultT]): ... # Valid StartT = TypeVar("StartT", default="StopT") # Swapped defaults around from previous example StopT = TypeVar("StopT", default=int) class slice(Generic[StartT, StopT, StepT]): ... # ^^^^^^ Invalid: ordering does not allow StopT to be bound Using a ``TypeVarLike`` from an outer scope as a default is not supported. Bound Rules ~~~~~~~~~~~ ``T2``'s bound must be a subtype of ``T1``'s bound. .. code-block:: py T = TypeVar("T", bound=float) TypeVar("Ok", default=T, bound=int) # Valid TypeVar("AlsoOk", default=T, bound=float) # Valid TypeVar("Invalid", default=T, bound=str) # Invalid: str is not a subtype of float Constraint Rules ~~~~~~~~~~~~~~~~ The constraints of ``T2`` must be a superset of the constraints of ``T1``. .. code-block:: py T1 = TypeVar("T1", bound=int) TypeVar("Invalid", float, str, default=T1) # Invalid: upper bound int is incompatible with constraints float or str T1 = TypeVar("T1", int, str) TypeVar("AlsoOk", int, str, bool, default=T1) # Valid TypeVar("AlsoInvalid", bool, complex, default=T1) # Invalid: {bool, complex} is not a superset of {int, str} ``TypeVarLike``\s as Parameters to Generics ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``TypeVarLike``\ s are valid as parameters to generics inside of a ``default`` when the first parameter is in scope as determined by the :ref:`previous section <696-scoping-rules>`. .. code-block:: py T = TypeVar("T") ListDefaultT = TypeVar("ListDefaultT", default=list[T]) class Bar(Generic[T, ListDefaultT]): def __init__(self, x: T, y: ListDefaultT): ... reveal_type(Bar) # type is type[Bar[T, ListDefaultT = list[T]]] reveal_type(Bar[int]) # type is type[Bar[int, list[int]]] reveal_type(Bar[int]()) # type is Bar[int, list[int]] reveal_type(Bar[int, list[str]]()) # type is Bar[int, list[str]] reveal_type(Bar[int, str]()) # type is Bar[int, str] Specialisation Rules ~~~~~~~~~~~~~~~~~~~~ ``TypeVarLike``\ s currently cannot be further subscripted. This might change if `Higher Kinded TypeVars `__ are implemented. ``Generic`` ``TypeAlias``\ es ''''''''''''''''''''''''''''' ``Generic`` ``TypeAlias``\ es should be able to be further subscripted following normal subscription rules. If a ``TypeVarLike`` has a default that hasn't been overridden it should be treated like it was substituted into the ``TypeAlias``. However, it can be specialised further down the line. .. code-block:: py class SomethingWithNoDefaults(Generic[T, T2]): ... MyAlias: TypeAlias = SomethingWithNoDefaults[int, DefaultStrT] # Valid reveal_type(MyAlias) # type is type[SomethingWithNoDefaults[int, DefaultStrT]] reveal_type(MyAlias[bool]()) # type is SomethingWithNoDefaults[int, bool] MyAlias[bool, int] # Invalid: too many arguments passed to MyAlias Subclassing ''''''''''' Subclasses of ``Generic``\ s with ``TypeVarLike``\ s that have defaults behave similarly to ``Generic`` ``TypeAlias``\ es. .. code-block:: py class SubclassMe(Generic[T, DefaultStrT]): x: DefaultStrT class Bar(SubclassMe[int, DefaultStrT]): ... reveal_type(Bar) # type is type[Bar[DefaultStrT = str]] reveal_type(Bar()) # type is Bar[str] reveal_type(Bar[bool]()) # type is Bar[bool] class Foo(SubclassMe[float]): ... reveal_type(Foo().x) # type is str Foo[str] # Invalid: Foo cannot be further subscripted class Baz(Generic[DefaultIntT, DefaultStrT]): ... class Spam(Baz): ... reveal_type(Spam()) # type is Using ``bound`` and ``default`` ''''''''''''''''''''''''''''''' If both ``bound`` and ``default`` are passed ``default`` must be a subtype of ``bound``. Otherwise the type checker should generate an error. .. code-block:: py TypeVar("Ok", bound=float, default=int) # Valid TypeVar("Invalid", bound=str, default=int) # Invalid: the bound and default are incompatible Constraints ''''''''''' For constrained ``TypeVar``\ s, the default needs to be one of the constraints. A type checker should generate an error even if it is a subtype of one of the constraints. .. code-block:: py TypeVar("Ok", float, str, default=float) # Valid TypeVar("Invalid", float, str, default=int) # Invalid: expected one of float or str got int .. _696-function-defaults: Function Defaults ''''''''''''''''' ``TypeVarLike``\ s currently are not supported in the signatures of functions as ensuring the ``default`` is returned in every code path where the ``TypeVarLike`` can go unsolved is too hard to implement. Implementation -------------- At runtime, this would involve the following changes to the ``typing`` module. - The classes ``TypeVar``, ``ParamSpec``, and ``TypeVarTuple`` should expose the type passed to ``default``. This would be available as a ``__default__`` attribute, which would be ``None`` if no argument is passed and ``NoneType`` if ``default=None``. The following changes would be required to both ``GenericAlias``\ es: - logic to determine the defaults required for a subscription. - ideally, logic to determine if subscription (like ``Generic[T, DefaultT]``) would be valid. A reference implementation of the runtime changes can be found at https://github.com/Gobot1234/cpython/tree/pep-696 A reference implementation of the type checker can be found at https://github.com/Gobot1234/mypy/tree/TypeVar-defaults Pyright currently supports this functionality. Interaction with PEP 695 ------------------------ If this PEP is accepted, the syntax proposed in :pep:`695` will be extended to introduce a way to specify defaults for type parameters using the "=" operator inside of the square brackets like so: .. code-block:: py # TypeVars class Foo[T = str]: ... # ParamSpecs class Baz[**P = [int, str]]: ... # TypeVarTuples class Qux[*Ts = *tuple[int, bool]]: ... # TypeAliases type Foo[T, U = str] = Bar[T, U] type Baz[**P = [int, str]] = Spam[**P] type Qux[*Ts = *tuple[str]] = Ham[*Ts] type Rab[U, T = str] = Bar[T, U] This functionality was included in the initial draft of :pep:`695` but was removed due to scope creep. Grammar Changes ''''''''''''''' :: type_param: | a=NAME b=[type_param_bound] d=[type_param_default] | a=NAME c=[type_param_constraint] d=[type_param_default] | '*' a=NAME d=[type_param_default] | '**' a=NAME d=[type_param_default] type_param_default: | '=' e=expression | '=' e=starred_expression This would mean that ``TypeVarLike``\ s with defaults proceeding those with non-defaults can be checked at compile time. Rejected Alternatives --------------------- Allowing the ``TypeVarLike``\s Defaults to Be Passed to ``type.__new__``'s ``**kwargs`` ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' .. code-block:: py T = TypeVar("T") @dataclass class Box(Generic[T], T=int): value: T | None = None While this is much easier to read and follows a similar rationale to the ``TypeVar`` `unary syntax `__, it would not be backwards compatible as ``T`` might already be passed to a metaclass/superclass or support classes that don't subclass ``Generic`` at runtime. Ideally, if :pep:`637` wasn't rejected, the following would be acceptable: .. code-block:: py T = TypeVar("T") @dataclass class Box(Generic[T = int]): value: T | None = None Allowing Non-defaults to Follow Defaults '''''''''''''''''''''''''''''''''''''''' .. code-block:: py YieldT = TypeVar("YieldT", default=Any) SendT = TypeVar("SendT", default=Any) ReturnT = TypeVar("ReturnT") class Coroutine(Generic[YieldT, SendT, ReturnT]): ... Coroutine[int] == Coroutine[Any, Any, int] Allowing non-defaults to follow defaults would alleviate the issues with returning types like ``Coroutine`` from functions where the most used type argument is the last (the return). Allowing non-defaults to follow defaults is too confusing and potentially ambiguous, even if only the above two forms were valid. Changing the argument order now would also break a lot of codebases. This is also solvable in most cases using a ``TypeAlias``. .. code-block:: py Coro: TypeAlias = Coroutine[Any, Any, T] Coro[int] == Coroutine[Any, Any, int] Having ``default`` Implicitly Be ``bound`` '''''''''''''''''''''''''''''''''''''''''' In an earlier version of this PEP, the ``default`` was implicitly set to ``bound`` if no value was passed for ``default``. This while convenient, could have a ``TypeVarLike`` with no default follow a ``TypeVarLike`` with a default. Consider: .. code-block:: py T = TypeVar("T", bound=int) # default is implicitly int U = TypeVar("U") class Foo(Generic[T, U]): ... # would expand to T = TypeVar("T", bound=int, default=int) U = TypeVar("U") class Foo(Generic[T, U]): ... This would have also been a breaking change for a small number of cases where the code relied on ``Any`` being the implicit default. Allowing ``TypeVarLike``\s with defaults to be used in function signatures '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' A previous version of this PEP allowed ``TypeVarLike``\s with defaults to be used in function signatures. This was removed for the reasons described in :ref:`696-function-defaults`. Hopefully, this can be added in the future if a way to get the runtime value of a type parameter is added. Allowing ``TypeVarLikes`` from outer scopes in ``default`` '''''''''''''''''''''''''''''''''''''''''''''''''''''''''' This was deemed too niche a feature to be worth the added complexity. If any cases arise where this is needed, it can be added in a future PEP. Acknowledgements ---------------- Thanks to the following people for their feedback on the PEP: Eric Traut, Jelle Zijlstra, Joshua Butt, Danny Yamamoto, Kaylynn Morgan and Jakub Kuczys Copyright --------- This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.