python-peps/peps/pep-0696.rst

572 lines
19 KiB
ReStructuredText

PEP: 696
Title: Type defaults for TypeVarLikes
Author: James Hilton-Balfe <gobot1234yt@gmail.com>
Sponsor: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://mail.python.org/archives/list/typing-sig@python.org/thread/7VWBZWXTCX6RAJO6GG67BAXUPFZ24NTC
Status: Draft
Type: Standards Track
Topic: Typing
Content-Type: text/x-rst
Created: 14-Jul-2022
Python-Version: 3.12
Post-History: `22-Mar-2022 <https://mail.python.org/archives/list/typing-sig@python.org/thread/7VWBZWXTCX6RAJO6GG67BAXUPFZ24NTC/>`__,
`08-Jan-2023 <https://discuss.python.org/t/pep-696-type-defaults-for-typevarlikes/22569/>`__,
Abstract
--------
This PEP introduces the concept of type defaults for
``TypeVarLike``\ s (``TypeVar``, ``ParamSpec`` and ``TypeVarTuple``),
which act as defaults for a type parameter when one is not specified or
the constraint solver isn't able to solve a type parameter to anything.
Default type argument support is available in some popular languages
such as C++, TypeScript, and Rust. A survey of type parameter syntax in
some common languages has been conducted by the author of :pep:`695`
and can be found in its
:pep:`Appendix A <695#appendix-a-survey-of-type-parameter-syntax>`.
Motivation
----------
.. code-block:: py
T = TypeVar("T", default=int) # This means that if no type is specified T = int
@dataclass
class Box(Generic[T]):
value: T | None = None
reveal_type(Box()) # type is Box[int]
reveal_type(Box(value="Hello World!")) # type is Box[str]
One place this `regularly comes
up <https://github.com/python/typing/issues/975>`__ is ``Generator``. I
propose changing the *stub definition* to something like:
.. code-block:: py
YieldT = TypeVar("YieldT")
SendT = TypeVar("SendT", default=None)
ReturnT = TypeVar("ReturnT", default=None)
class Generator(Generic[YieldT, SendT, ReturnT]): ...
Generator[int] == Generator[int, None] == Generator[int, None, None]
This is also useful for a ``Generic`` that is commonly over one type.
.. code-block:: py
class Bot: ...
BotT = TypeVar("BotT", bound=Bot, default=Bot)
class Context(Generic[BotT]):
bot: BotT
class MyBot(Bot): ...
reveal_type(Context().bot) # type is Bot # notice this is not Any which is what it would be currently
reveal_type(Context[MyBot]().bot) # type is MyBot
Not only does this improve typing for those who explicitly use it, it
also helps non-typing users who rely on auto-complete to speed up their
development.
This design pattern is common in projects like:
- `discord.py <https://github.com/Rapptz/discord.py>`__ — where the
example above was taken from.
- `NumPy <https://github.com/numpy/numpy>`__ — the default for types
like ``ndarray``'s ``dtype`` would be ``float64``. Currently it's
``Unknown`` or ``Any``.
- `TensorFlow <https://github.com/tensorflow/tensorflow>`__ — this
could be used for Tensor similarly to ``numpy.ndarray`` and would be
useful to simplify the definition of ``Layer``.
Specification
-------------
Default Ordering and Subscription Rules
'''''''''''''''''''''''''''''''''''''''
The order for defaults should follow the standard function parameter
rules, so a ``TypeVarLike`` with no ``default`` cannot follow one with
a ``default`` value. Doing so should ideally raise a ``TypeError`` in
``typing._GenericAlias``/``types.GenericAlias``, and a type checker
should flag this an error.
.. code-block:: py
DefaultStrT = TypeVar("DefaultStrT", default=str)
DefaultIntT = TypeVar("DefaultIntT", default=int)
DefaultBoolT = TypeVar("DefaultBoolT", default=bool)
T = TypeVar("T")
T2 = TypeVar("T2")
class NonDefaultFollowsDefault(Generic[DefaultStrT, T]): ... # Invalid: non-default TypeVars cannot follow ones with defaults
class NoNonDefaults(Generic[DefaultStrT, DefaultIntT]): ...
(
NoNoneDefaults ==
NoNoneDefaults[str] ==
NoNoneDefaults[str, int]
) # All valid
class OneDefault(Generic[T, DefaultBoolT]): ...
OneDefault[float] == OneDefault[float, bool] # Valid
reveal_type(OneDefault) # type is type[OneDefault[T, DefaultBoolT = bool]]
reveal_type(OneDefault[float]()) # type is OneDefault[float, bool]
class AllTheDefaults(Generic[T1, T2, DefaultStrT, DefaultIntT, DefaultBoolT]): ...
reveal_type(AllTheDefaults) # type is type[AllTheDefaults[T1, T2, DefaultStrT = str, DefaultIntT = int, DefaultBoolT = bool]]
reveal_type(AllTheDefaults[int, complex]()) # type is AllTheDefaults[int, complex, str, int, bool]
AllTheDefaults[int] # Invalid: expected 2 arguments to AllTheDefaults
(
AllTheDefaults[int, complex] ==
AllTheDefaults[int, complex, str] ==
AllTheDefaults[int, complex, str, int] ==
AllTheDefaults[int, complex, str, int, bool]
) # All valid
This cannot be enforced at runtime for functions, for now, but in the
future, this might be possible (see `Interaction with PEP
695 <#interaction-with-pep-695>`__).
``ParamSpec`` Defaults
''''''''''''''''''''''
``ParamSpec`` defaults are defined using the same syntax as
``TypeVar`` \ s but use a ``list`` of types or an ellipsis
literal "``...``" or another in-scope ``ParamSpec`` (see `Scoping Rules`_).
.. code-block:: py
DefaultP = ParamSpec("DefaultP", default=[str, int])
class Foo(Generic[DefaultP]): ...
reveal_type(Foo) # type is type[Foo[DefaultP = [str, int]]]
reveal_type(Foo()) # type is Foo[[str, int]]
reveal_type(Foo[[bool, bool]]()) # type is Foo[[bool, bool]]
``TypeVarTuple`` Defaults
'''''''''''''''''''''''''
``TypeVarTuple`` defaults are defined using the same syntax as
``TypeVar`` \ s but use an unpacked tuple of types instead of a single type
or another in-scope ``TypeVarTuple`` (see `Scoping Rules`_).
.. code-block:: py
DefaultTs = TypeVarTuple("DefaultTs", default=Unpack[tuple[str, int]])
class Foo(Generic[*DefaultTs]): ...
reveal_type(Foo) # type is type[Foo[DefaultTs = *tuple[str, int]]]
reveal_type(Foo()) # type is Foo[str, int]
reveal_type(Foo[int, bool]()) # type is Foo[int, bool]
Using Another ``TypeVarLike`` as ``default``
''''''''''''''''''''''''''''''''''''''''''''
This allows for a value to be used again when the constraints solver
fails to solve a constraint for a type, or the type parameter to a
generic is missing but another type parameter is specified.
To use another ``TypeVarLike`` as a default the ``default`` and the
``TypeVarLike`` must be the same type (a ``TypeVar``'s default must be
a ``TypeVar``, etc.).
`This could be used on builtins.slice <https://github.com/python/typing/issues/159>`__
where the ``start`` parameter should default to ``int``, ``stop``
default to the type of ``start`` and step default to ``int | None``.
.. code-block:: py
StartT = TypeVar("StartT", default=int)
StopT = TypeVar("StopT", default=StartT)
StepT = TypeVar("StepT", default=int | None)
class slice(Generic[StartT, StopT, StepT]): ...
reveal_type(slice) # type is type[slice[StartT = int, StopT = StartT, StepT = int | None]]
reveal_type(slice()) # type is slice[int, int, int | None]
reveal_type(slice[str]()) # type is slice[str, str, int | None]
reveal_type(slice[str, bool, timedelta]()) # type is slice[str, bool, timedelta]
T2 = TypeVar("T2", default=DefaultStrT)
class Foo(Generic[DefaultStrT, T2]):
def __init__(self, a: DefaultStrT, b: T2) -> None: ...
reveal_type(Foo(1, "")) # type is Foo[int, str]
Foo[int](1, "") # Invalid: Foo[int, str] cannot be assigned to self: Foo[int, int] in Foo.__init__
Foo[int]("", 1) # Invalid: Foo[str, int] cannot be assigned to self: Foo[int, int] in Foo.__init__
When using a ``TypeVarLike`` as the default to another ``TypeVarLike``.
Where ``T1`` is the default for ``T2`` the following rules apply.
``TypeVarTuple``\s are not supported because:
- `Scoping Rules`_ does not allow usage of ``TypeVarLikes``
from outer scopes.
- Multiple ``TypeVarTuple``\s cannot appear in the type
parameter list for a single class, as specified in
:pep:`646#multiple-type-variable-tuples-not-allowed`.
- ``TypeVarLike`` defaults in functions are not supported.
These reasons leave no current valid location where a
``TypeVarTuple`` could have a default.
Scoping Rules
~~~~~~~~~~~~~
``T1`` must be used before ``T2`` in the parameter list of the generic.
.. code-block:: py
DefaultT = TypeVar("DefaultT", default=T)
class Foo(Generic[T, DefaultT]): ... # Valid
class Foo(Generic[T]):
class Bar(Generic[DefaultT]): ... # Valid
StartT = TypeVar("StartT", default="StopT") # Swapped defaults around from previous example
StopT = TypeVar("StopT", default=int)
class slice(Generic[StartT, StopT, StepT]): ...
# ^^^^^^ Invalid: ordering does not allow StopT to be bound
Using a ``TypeVarLike`` from an outer scope as a default is not supported.
Bound Rules
~~~~~~~~~~~
``T2``'s bound must be a subtype of ``T1``'s bound.
.. code-block:: py
T = TypeVar("T", bound=float)
TypeVar("Ok", default=T, bound=int) # Valid
TypeVar("AlsoOk", default=T, bound=float) # Valid
TypeVar("Invalid", default=T, bound=str) # Invalid: str is not a subtype of float
Constraint Rules
~~~~~~~~~~~~~~~~
The constraints of ``T2`` must be a superset of the constraints of ``T1``.
.. code-block:: py
T1 = TypeVar("T1", bound=int)
TypeVar("Invalid", float, str, default=T1) # Invalid: upper bound int is incompatible with constraints float or str
T1 = TypeVar("T1", int, str)
TypeVar("AlsoOk", int, str, bool, default=T1) # Valid
TypeVar("AlsoInvalid", bool, complex, default=T1) # Invalid: {bool, complex} is not a superset of {int, str}
``TypeVarLike``\s as Parameters to Generics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``TypeVarLike``\ s are valid as parameters to generics inside of a
``default`` when the first parameter is in scope as determined by the
`previous section <scoping rules_>`_.
.. code-block:: py
T = TypeVar("T")
ListDefaultT = TypeVar("ListDefaultT", default=list[T])
class Bar(Generic[T, ListDefaultT]):
def __init__(self, x: T, y: ListDefaultT): ...
reveal_type(Bar) # type is type[Bar[T, ListDefaultT = list[T]]]
reveal_type(Bar[int]) # type is type[Bar[int, list[int]]]
reveal_type(Bar[int]()) # type is Bar[int, list[int]]
reveal_type(Bar[int, list[str]]()) # type is Bar[int, list[str]]
reveal_type(Bar[int, str]()) # type is Bar[int, str]
Specialisation Rules
~~~~~~~~~~~~~~~~~~~~
``TypeVarLike``\ s currently cannot be further subscripted. This might
change if `Higher Kinded TypeVars <https://github.com/python/typing/issues/548>`__
are implemented.
``Generic`` ``TypeAlias``\ es
'''''''''''''''''''''''''''''
``Generic`` ``TypeAlias``\ es should be able to be further subscripted
following normal subscription rules. If a ``TypeVarLike`` has a default
that hasn't been overridden it should be treated like it was
substituted into the ``TypeAlias``. However, it can be specialised
further down the line.
.. code-block:: py
class SomethingWithNoDefaults(Generic[T, T2]): ...
MyAlias: TypeAlias = SomethingWithNoDefaults[int, DefaultStrT] # Valid
reveal_type(MyAlias) # type is type[SomethingWithNoDefaults[int, DefaultStrT]]
reveal_type(MyAlias[bool]()) # type is SomethingWithNoDefaults[int, bool]
MyAlias[bool, int] # Invalid: too many arguments passed to MyAlias
Subclassing
'''''''''''
Subclasses of ``Generic``\ s with ``TypeVarLike``\ s that have defaults
behave similarly to ``Generic`` ``TypeAlias``\ es.
.. code-block:: py
class SubclassMe(Generic[T, DefaultStrT]):
x: DefaultStrT
class Bar(SubclassMe[int, DefaultStrT]): ...
reveal_type(Bar) # type is type[Bar[DefaultStrT = str]]
reveal_type(Bar()) # type is Bar[str]
reveal_type(Bar[bool]()) # type is Bar[bool]
class Foo(SubclassMe[float]): ...
reveal_type(Foo().x) # type is str
Foo[str] # Invalid: Foo cannot be further subscripted
class Baz(Generic[DefaultIntT, DefaultStrT]): ...
class Spam(Baz): ...
reveal_type(Spam()) # type is <subclass of Baz[int, str]>
Using ``bound`` and ``default``
'''''''''''''''''''''''''''''''
If both ``bound`` and ``default`` are passed ``default`` must be a
subtype of ``bound``. Otherwise the type checker should generate an
error.
.. code-block:: py
TypeVar("Ok", bound=float, default=int) # Valid
TypeVar("Invalid", bound=str, default=int) # Invalid: the bound and default are incompatible
Constraints
'''''''''''
For constrained ``TypeVar``\ s, the default needs to be one of the
constraints. A type checker should generate an error even if it is a
subtype of one of the constraints.
.. code-block:: py
TypeVar("Ok", float, str, default=float) # Valid
TypeVar("Invalid", float, str, default=int) # Invalid: expected one of float or str got int
.. _696-function-defaults:
Function Defaults
'''''''''''''''''
``TypeVarLike``\ s currently are not supported in the signatures of
functions as ensuring the ``default`` is returned in every code path
where the ``TypeVarLike`` can go unsolved is too hard to implement.
Implementation
--------------
At runtime, this would involve the following changes to the ``typing``
module.
- The classes ``TypeVar``, ``ParamSpec``, and ``TypeVarTuple`` should
expose the type passed to ``default``. This would be available as
a ``__default__`` attribute, which would be ``None`` if no argument
is passed and ``NoneType`` if ``default=None``.
The following changes would be required to both ``GenericAlias``\ es:
- logic to determine the defaults required for a subscription.
- ideally, logic to determine if subscription (like
``Generic[T, DefaultT]``) would be valid.
A reference implementation of the runtime changes can be found at
https://github.com/Gobot1234/cpython/tree/pep-696
A reference implementation of the type checker can be found at
https://github.com/Gobot1234/mypy/tree/TypeVar-defaults
Pyright currently supports this functionality.
Interaction with PEP 695
------------------------
The syntax proposed in :pep:`695` will be extended to introduce a way
to specify defaults for type parameters using the "=" operator inside
of the square brackets like so:
.. code-block:: py
# TypeVars
class Foo[T = str]: ...
# ParamSpecs
class Baz[**P = [int, str]]: ...
# TypeVarTuples
class Qux[*Ts = *tuple[int, bool]]: ...
# TypeAliases
type Foo[T, U = str] = Bar[T, U]
type Baz[**P = [int, str]] = Spam[**P]
type Qux[*Ts = *tuple[str]] = Ham[*Ts]
type Rab[U, T = str] = Bar[T, U]
:ref:`Similarly to the bound for a type parameter <695-scoping-behavior>`,
defaults should be lazily evaluated, with the same scoping rules to
avoid the unnecessary usage of quotes around them.
This functionality was included in the initial draft of :pep:`695` but
was removed due to scope creep.
Grammar Changes
'''''''''''''''
::
type_param:
| a=NAME b=[type_param_bound] d=[type_param_default]
| a=NAME c=[type_param_constraint] d=[type_param_default]
| '*' a=NAME d=[type_param_default]
| '**' a=NAME d=[type_param_default]
type_param_default:
| '=' e=expression
| '=' e=starred_expression
This would mean that ``TypeVarLike``\ s with defaults proceeding those
with non-defaults can be checked at compile time.
Rejected Alternatives
---------------------
Allowing the ``TypeVarLike``\s Defaults to Be Passed to ``type.__new__``'s ``**kwargs``
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
.. code-block:: py
T = TypeVar("T")
@dataclass
class Box(Generic[T], T=int):
value: T | None = None
While this is much easier to read and follows a similar rationale to the
``TypeVar`` `unary
syntax <https://github.com/python/typing/issues/813>`__, it would not be
backwards compatible as ``T`` might already be passed to a
metaclass/superclass or support classes that don't subclass ``Generic``
at runtime.
Ideally, if :pep:`637` wasn't rejected, the following would be acceptable:
.. code-block:: py
T = TypeVar("T")
@dataclass
class Box(Generic[T = int]):
value: T | None = None
Allowing Non-defaults to Follow Defaults
''''''''''''''''''''''''''''''''''''''''
.. code-block:: py
YieldT = TypeVar("YieldT", default=Any)
SendT = TypeVar("SendT", default=Any)
ReturnT = TypeVar("ReturnT")
class Coroutine(Generic[YieldT, SendT, ReturnT]): ...
Coroutine[int] == Coroutine[Any, Any, int]
Allowing non-defaults to follow defaults would alleviate the issues with
returning types like ``Coroutine`` from functions where the most used
type argument is the last (the return). Allowing non-defaults to follow
defaults is too confusing and potentially ambiguous, even if only the
above two forms were valid. Changing the argument order now would also
break a lot of codebases. This is also solvable in most cases using a
``TypeAlias``.
.. code-block:: py
Coro: TypeAlias = Coroutine[Any, Any, T]
Coro[int] == Coroutine[Any, Any, int]
Having ``default`` Implicitly Be ``bound``
''''''''''''''''''''''''''''''''''''''''''
In an earlier version of this PEP, the ``default`` was implicitly set
to ``bound`` if no value was passed for ``default``. This while
convenient, could have a ``TypeVarLike`` with no default follow a
``TypeVarLike`` with a default. Consider:
.. code-block:: py
T = TypeVar("T", bound=int) # default is implicitly int
U = TypeVar("U")
class Foo(Generic[T, U]):
...
# would expand to
T = TypeVar("T", bound=int, default=int)
U = TypeVar("U")
class Foo(Generic[T, U]):
...
This would have also been a breaking change for a small number of cases
where the code relied on ``Any`` being the implicit default.
Allowing ``TypeVarLike``\s with defaults to be used in function signatures
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
A previous version of this PEP allowed ``TypeVarLike``\s with defaults to be used in
function signatures. This was removed for the reasons described in
`Function Defaults`_. Hopefully, this can be added in the future if
a way to get the runtime value of a type parameter is added.
Allowing ``TypeVarLikes`` from outer scopes in ``default``
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
This was deemed too niche a feature to be worth the added complexity.
If any cases arise where this is needed, it can be added in a future PEP.
Acknowledgements
----------------
Thanks to the following people for their feedback on the PEP:
Eric Traut, Jelle Zijlstra, Joshua Butt, Danny Yamamoto, Kaylynn Morgan
and Jakub Kuczys
Copyright
---------
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.