python-peps/pep-0696.rst

PEP: 696
Title: Type defaults for TypeVarLikes
Author: James Hilton-Balfe <gobot1234yt@gmail.com>
Sponsor: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: typing-sig@python.org
Status: Draft
Type: Standards Track
Topic: Typing
Content-Type: text/x-rst
Created: 14-Jul-2022
Python-Version: 3.12

Abstract
--------

This PEP introduces the concept of type defaults for
``TypeVarLike``\ s (``TypeVar``, ``ParamSpec`` and ``TypeVarTuple``),
which act as defaults for a type parameter when one is not specified or
the constraint solver isn't able to solve a type parameter to anything.

Default type argument support is available in some popular languages
such as C++, TypeScript, and Rust. A survey of type parameter syntax in
some common languages has been conducted by the author of :pep:`695`
and can be found in its
:pep:`Appendix A <695#appendix-a-survey-of-type-parameter-syntax>`.


Motivation
----------

.. code-block:: py

   T = TypeVar("T", default=int)  # This means that if no type is specified T = int

   @dataclass
   class Box(Generic[T]):
       value: T | None = None

   reveal_type(Box())                      # type is Box[int]
   reveal_type(Box(value="Hello World!"))  # type is Box[str]

One place this `regularly comes
up <https://github.com/python/typing/issues/975>`__ is ``Generator``. I
propose changing the *stub definition* to something like:

.. code-block:: py

   YieldT = TypeVar("YieldT")
   SendT = TypeVar("SendT", default=None)
   ReturnT = TypeVar("ReturnT", default=None)

   class Generator(Generic[YieldT, SendT, ReturnT]): ...

   Generator[int] == Generator[int, None] == Generator[int, None, None]

This is also useful for a ``Generic`` that is commonly over one type.

.. code-block:: py

   class Bot: ...

   BotT = TypeVar("BotT", bound=Bot, default=Bot)

   class Context(Generic[BotT]):
       bot: BotT

   class MyBot(Bot): ...

   reveal_type(Context().bot)         # type is Bot  # notice this is not Any which is what it would be currently
   reveal_type(Context[MyBot]().bot)  # type is MyBot

Not only does this improve typing for those who explicitly use it, it
also helps non-typing users who rely on auto-complete to speed up their
development.

This design pattern is common in projects like:
 - `discord.py <https://github.com/Rapptz/discord.py>`__ — where the
   example above was taken from.
 - `NumPy <https://github.com/numpy/numpy>`__ — the default for types
   like ``ndarray``'s ``dtype`` would be ``float64``. Currently it's
   ``Unknown`` or ``Any``.
 - `TensorFlow <https://github.com/tensorflow/tensorflow>`__ — this
   could be used for Tensor similarly to ``numpy.ndarray`` and would be
   useful to simplify the definition of ``Layer``.


Specification
-------------

Default Ordering and Subscription Rules
'''''''''''''''''''''''''''''''''''''''

The order for defaults should follow the standard function parameter
rules, so a ``TypeVarLike`` with no ``default`` cannot follow one with
a ``default`` value. Doing so should ideally raise a ``TypeError`` in
``typing._GenericAlias``/``types.GenericAlias``, and a type checker
should flag this an error.

.. code-block:: py

   DefaultStrT = TypeVar("DefaultStrT", default=str)
   DefaultIntT = TypeVar("DefaultIntT", default=int)
   DefaultBoolT = TypeVar("DefaultBoolT", default=bool)
   T = TypeVar("T")
   T2 = TypeVar("T2")

   class NonDefaultFollowsDefault(Generic[DefaultStrT, T]): ...  # Invalid: non-default TypeVars cannot follow ones with defaults


   class NoNonDefaults(Generic[DefaultStrT, DefaultIntT]): ...

   (
       NoNoneDefaults ==
       NoNoneDefaults[str] ==
       NoNoneDefaults[str, int]
   )  # All valid


   class OneDefault(Generic[T, DefaultBoolT]): ...

   OneDefault[float] == OneDefault[float, bool]  # Valid


   class AllTheDefaults(Generic[T1, T2, DefaultStrT, DefaultIntT, DefaultBoolT]): ...

   AllTheDefaults[int]  # Invalid: expected 2 arguments to AllTheDefaults
   (
       AllTheDefaults[int, complex] ==
       AllTheDefaults[int, complex, str] ==
       AllTheDefaults[int, complex, str, int] ==
       AllTheDefaults[int, complex, str, int, bool]
   )  # All valid

This cannot be enforced at runtime for functions, for now, but in the
future, this might be possible (see `Interaction with PEP
695 <#interaction-with-pep-695>`__).

``ParamSpec`` Defaults
''''''''''''''''''''''

``ParamSpec`` defaults are defined using the same syntax as
``TypeVar`` \ s but use a ``list`` or ``tuple`` of types or an ellipsis
literal "``...``".

.. code-block:: py

   DefaultP = ParamSpec("DefaultP", default=(str, int))

   class Foo(Generic[DefaultP]): ...

   reveal_type(Foo())                # type is Foo[(str, int)]
   reveal_type(Foo[(bool, bool)]())  # type is Foo[(bool, bool)]

``TypeVarTuple`` Defaults
'''''''''''''''''''''''''

``TypeVarTuple`` defaults are defined using the same syntax as
``TypeVar`` \ s but use an unpacked tuple of types instead of a single type.

.. code-block:: py

   DefaultTs = TypeVarTuple("DefaultTs", default=Unpack[tuple[str, int]])

   class Foo(Generic[*DefaultTs]): ...

   reveal_type(Foo())             # type is Foo[str, int]
   reveal_type(Foo[int, bool]())  # type is Foo[int, bool]

Using Another ``TypeVarLike`` as ``default``
''''''''''''''''''''''''''''''''''''''''''''

This allows for a value to be used again when the constraints solver
fails to solve a constraint for a type, or the type parameter to a
generic is missing but another type parameter is specified.

To use another ``TypeVarLike`` as a default the ``default`` and the
``TypeVarLike`` must be the same type (a ``TypeVar``'s default must be
a ``TypeVar``, etc.).

`This could be used on builtins.slice <https://github.com/python/typing/issues/159>`__
where the ``start`` parameter should default to ``int``, ``stop``
default to the type of ``start`` and step default to ``int | None``.

.. code-block:: py

   StartT = TypeVar("StartT", default=int)
   StopT = TypeVar("StopT", default=StartT)
   StepT = TypeVar("StepT", default=int | None)

   class slice(Generic[StartT, StopT, StepT]): ...

   reveal_type(slice())                        # type is slice[int, int, int | None]
   reveal_type(slice[str]())                   # type is slice[str, str, int | None]
   reveal_type(slice[str, bool, timedelta]())  # type is slice[str, bool, timedelta]

When using a ``TypeVarLike`` as the default to another ``TypeVarLike``.
Where ``T1`` is the default for ``T2`` the following rules apply.

.. _scoping-rules:

Scoping Rules
~~~~~~~~~~~~~

``T1`` must be used before ``T2`` in the parameter list of the generic,
or be bound in an outer class or function scope.

.. code-block:: py

   DefaultT = TypeVar("DefaultT", default=T)

   class Foo(Generic[T, DefaultT]): ...   # Valid
   def bar(x: T, y: DefaultT): ...        # Valid
   class Foo(Generic[T]):
       class Bar(Generic[DefaultT]): ...  # Valid
   def outer(x: T):
       def inner(y: DefaultT): ...        # Valid

   StartT = TypeVar("StartT", default="StopT")  # Swapped defaults around from previous example
   StopT = TypeVar("StopT", default=int)
   class slice(Generic[StartT, StopT, StepT]): ...
                     # ^^^^^^ Invalid: ordering does not allow StopT to be bound
   def baz(x: DefaultT, y: T): ...
            # ^^^^^^^^ Invalid: ordering does not allow DefaultT to be bound

Bound Rules
~~~~~~~~~~~

``T2``'s bound must be a subtype of ``T1``'s bound.

.. code-block:: py

   T = TypeVar("T", bound=float)
   TypeVar("Ok", default=T, bound=int)        # Valid
   TypeVar("AlsoOk", default=T, bound=float)  # Valid
   TypeVar("Invalid", default=T, bound=str)   # Invalid: str is not a subtype of float

Constraint Rules
~~~~~~~~~~~~~~~~

The constraints of ``T2`` must be a superset of the constraints of ``T1``.

.. code-block:: py

   T1 = TypeVar("T1", bound=int)
   TypeVar("Invalid", float, str, default=T1)         # Invalid: upper bound int is incompatible with constraints float or str

   T1 = TypeVar("T1", int, str)
   TypeVar("AlsoOk", int, str, bool, default=T1)      # Valid
   TypeVar("AlsoInvalid", bool, complex, default=T1)  # Invalid: {bool, complex} is not a superset of {int, str}


``TypeVarLike``\s as Parameters to Generics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``TypeVarLike``\ s are valid as parameters to generics inside of a
``default`` when the first parameter is in scope as determined by the
:ref:`previous section <scoping-rules>`.

.. code-block:: py

   T = TypeVar("T")
   ListDefaultT = TypeVar("ListDefaultT", default=list[T])

   class Bar(Generic[T, ListDefaultT]):
       def __init__(self, x: T, y: ListDefaultT): ...

   reveal_type(Bar[int])             # type is Bar[int, list[int]]
   reveal_type(Bar[int, list[str]])  # type is Bar[int, list[str]]
   reveal_type(Bar[int, str])        # type is Bar[int, str]

Specialisation Rules
~~~~~~~~~~~~~~~~~~~~

``TypeVarLike``\ s currently cannot be further subscripted. This might
change if `Higher Kinded TypeVars <https://github.com/python/typing/issues/548>`__
are implemented.


``Generic`` ``TypeAlias``\ es
'''''''''''''''''''''''''''''

``Generic`` ``TypeAlias``\ es should be able to be further subscripted
following normal subscription rules. If a ``TypeVarLike`` has a default
that hasn't been overridden it should be treated like it was
substituted into the ``TypeAlias``. However, it can be specialised
further down the line.

.. code-block:: py

   class SomethingWithNoDefaults(Generic[T, T2]): ...

   MyAlias: TypeAlias = SomethingWithNoDefaults[int, DefaultStrT]  # Valid
   reveal_type(MyAlias())        # type is SomethingWithNoDefaults[int, str]
   reveal_type(MyAlias[bool]())  # type is SomethingWithNoDefaults[int, bool]

   MyAlias[bool, int]  # Invalid: too many arguments passed to MyAlias

Subclassing
'''''''''''

Subclasses of ``Generic``\ s with ``TypeVarLike``\ s that have defaults
behave similarly to ``Generic`` ``TypeAlias``\ es.

.. code-block:: py

   class SubclassMe(Generic[T, DefaultStrT]):
       x: DefaultStrT

   class Bar(SubclassMe[int, DefaultStrT]): ...
   reveal_type(Bar())        # type is Bar[str]
   reveal_type(Bar[bool]())  # type is Bar[bool]

   class Foo(SubclassMe[float]): ...

   reveal_type(Foo().x)  # type is str

   Foo[str]  # Invalid: Foo cannot be further subscripted

   class Baz(Generic[DefaultIntT, DefaultStrT]): ...

   class Spam(Baz): ...
   reveal_type(Spam())  # type is <subclass of Baz[int, str]>

Using ``bound`` and ``default``
'''''''''''''''''''''''''''''''

If both ``bound`` and ``default`` are passed ``default`` must be a
subtype of ``bound``. Otherwise the type checker should generate an
error.

.. code-block:: py

   TypeVar("Ok", bound=float, default=int)     # Valid
   TypeVar("Invalid", bound=str, default=int)  # Invalid: the bound and default are incompatible

Constraints
'''''''''''

For constrained ``TypeVar``\ s, the default needs to be one of the
constraints. A type checker should generate an error even if it is a
subtype of one of the constraints.

.. code-block:: py

   TypeVar("Ok", float, str, default=float)     # Valid
   TypeVar("Invalid", float, str, default=int)  # Invalid: expected one of float or str got int

Function Defaults
'''''''''''''''''

``TypeVarLike``\ s currently can only be used where a parameter can go unsolved.

.. code-block:: py

   def foo(a: DefaultStrT | None = None) -> DefaultStrT: ...

   reveal_type(foo(1)) # type is int
   reveal_type(foo())  # type is str

If they are used where the parameter type is known, the defaults
should just be ignored and a type checker can emit a warning.


Implementation
--------------

At runtime, this would involve the following changes to the ``typing``
module.

- a new class ``_DefaultMixin`` needs to be added which is a superclass
  of ``TypeVar``, ``ParamSpec``, and ``TypeVarTuple``.

   -  the type passed to default would be available as a ``__default__``
      attribute and a sentinel would need to be added that can be
      checked against to see if it was supplied.

The following changes would be required to both ``GenericAlias``\ es:

-  logic to determine the defaults required for a subscription.
-  ideally, logic to determine if subscription (like
   ``Generic[T, DefaultT]``) would be valid.

A reference implementation of the type checker can be found at
https://github.com/Gobot1234/mypy/tree/TypeVar-defaults


Interaction with PEP 695
------------------------

If this PEP is accepted, the syntax proposed in :pep:`695` will be
extended to introduce a way to specify defaults for type parameters
using the "=" operator inside of the square brackets like so:

.. code-block:: py

   # TypeVars
   class Foo[T = str]: ...
   def bar[U = int](): ...

   # ParamSpecs
   class Baz[**P = (int, str)]: ...
   def spam[**Q = (bool,)](): ...

   # TypeVarTuples
   class Qux[*Ts = *tuple[int, bool]]: ...
   def ham[*Us = *tuple[str]](): ...

   # TypeAliases
   type Foo[T, U = str] = Bar[T, U]
   type Baz[**P = (int, str)] = Spam[**P]
   type Qux[*Ts = *tuple[str]] = Ham[*Ts]
   type Rab[U, T = str] = Bar[T, U]

This functionality was included in the initial draft of :pep:`695` but
was removed due to scope creep.

Grammar Changes
'''''''''''''''

::

    type_param:
        | a=NAME b=[type_param_bound] d=[type_param_default]
        | a=NAME c=[type_param_constraint] d=[type_param_default]
        | '*' a=NAME d=[type_param_default]
        | '**' a=NAME d=[type_param_default]

    type_param_default:
        | '=' e=expression
        | '=' e=starred_expression

This would mean that ``TypeVarLike``\ s with defaults proceeding those
with non-defaults can be checked at compile time.


Rejected Alternatives
---------------------

Allowing the ``TypeVarLike``\s Defaults to Be Passed to ``type.__new__``'s ``**kwargs``
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

.. code-block:: py

   T = TypeVar("T")

   @dataclass
   class Box(Generic[T], T=int):
       value: T | None = None

While this is much easier to read and follows a similar rationale to the
``TypeVar`` `unary
syntax <https://github.com/python/typing/issues/813>`__, it would not be
backwards compatible as ``T`` might already be passed to a
metaclass/superclass or support classes that don't subclass ``Generic``
at runtime.

Ideally, if :pep:`637` wasn't rejected, the following would be acceptable:

.. code-block:: py

   T = TypeVar("T")

   @dataclass
   class Box(Generic[T = int]):
       value: T | None = None

Allowing Non-defaults to Follow Defaults
''''''''''''''''''''''''''''''''''''''''

.. code-block:: py

   YieldT = TypeVar("YieldT", default=Any)
   SendT = TypeVar("SendT", default=Any)
   ReturnT = TypeVar("ReturnT")

   class Coroutine(Generic[YieldT, SendT, ReturnT]): ...

   Coroutine[int] == Coroutine[Any, Any, int]

Allowing non-defaults to follow defaults would alleviate the issues with
returning types like ``Coroutine`` from functions where the most used
type argument is the last (the return). Allowing non-defaults to follow
defaults is too confusing and potentially ambiguous, even if only the
above two forms were valid. Changing the argument order now would also
break a lot of codebases. This is also solvable in most cases using a
``TypeAlias``.

.. code-block:: py

   Coro: TypeAlias = Coroutine[Any, Any, T]
   Coro[int] == Coroutine[Any, Any, int]

Having ``default`` Implicitly Be ``bound``
''''''''''''''''''''''''''''''''''''''''''

In an earlier version of this PEP, the ``default`` was implicitly set
to ``bound`` if no value was passed for ``default``. This while
convenient, could have a ``TypeVarLike`` with no default follow a
``TypeVarLike`` with a default. Consider:

.. code-block:: py

   T = TypeVar("T", bound=int)  # default is implicitly int
   U = TypeVar("U")

   class Foo(Generic[T, U]):
       ...

   # would expand to

   T = TypeVar("T", bound=int, default=int)
   U = TypeVar("U")

   class Foo(Generic[T, U]):
       ...

This would have also been a breaking change for a small number of cases
where the code relied on ``Any`` being the implicit default.


Acknowledgements
----------------

Thanks to the following people for their feedback on the PEP:

Eric Traut, Jelle Zijlstra, Joshua Butt, Danny Yamamoto, Kaylynn Morgan
and Jakub Kuczys


Copyright
---------
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.