1280 lines
43 KiB
ReStructuredText
1280 lines
43 KiB
ReStructuredText
PEP: 695
|
|
Title: Type Parameter Syntax
|
|
Author: Eric Traut <erictr at microsoft.com>
|
|
Sponsor: Guido van Rossum <guido@python.org>
|
|
Discussions-To: typing-sig@python.org
|
|
Status: Draft
|
|
Type: Standards Track
|
|
Content-Type: text/x-rst
|
|
Created: 15-Jun-2022
|
|
Python-Version: 3.12
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
This PEP specifies an improved syntax for specifying type parameters within
|
|
a generic class, function, or type alias. It also introduces a new statement
|
|
for declaring type aliases.
|
|
|
|
|
|
Motivation
|
|
==========
|
|
|
|
:pep:`484` introduced type variables into the language. :pep:`612` built
|
|
upon this concept by introducing parameter specifications, and
|
|
:pep:`646` added variadic type variables.
|
|
|
|
While generic types and type parameters have grown in popularity, the
|
|
syntax for specifying type parameters still feels "bolted on" to Python.
|
|
This is a source of confusion among Python developers.
|
|
|
|
There is consensus within the Python static typing community that it is time
|
|
to provide a formal syntax and bring Python in alignment with other modern
|
|
programming languages that support generic types.
|
|
|
|
An analysis of 25 popular typed Python libraries revealed that type
|
|
variables (in particular, the ``typing.TypeVar`` symbol) were used in
|
|
14% of modules. This percentage is likely to increase if type parameters
|
|
become less cumbersome to use.
|
|
|
|
|
|
Points of Confusion
|
|
-------------------
|
|
|
|
While the use of type variables has become widespread, the manner in which
|
|
they are specified within code is the source of confusion among many
|
|
Python developers. There are a couple of factors that contribute to this
|
|
confusion.
|
|
|
|
The scoping rules for type variables are difficult to understand. Type
|
|
variables are typically allocated within the global scope, but their semantic
|
|
meaning is valid only when used within the context of a generic class,
|
|
function or or type alias. A single runtime instance of a type variable may be
|
|
reused in multiple generic contexts, and it has a different semantic meaning
|
|
in each of these contexts. This PEP proposes to eliminate this source of
|
|
confusion by declaring type parameters at a natural place within a class,
|
|
function or type alias declaration statement.
|
|
|
|
Generic type aliases are often misused because it is not clear to developers
|
|
that a type argument must be supplied when the type alias is used. This leads
|
|
to an implied type argument of ``Any``, which is rarely the intent. This PEP
|
|
proposes to add new syntax that makes generic type alias declarations
|
|
clear.
|
|
|
|
:pep:`483` and :pep:`484` introduced the concept of "variance" for a type
|
|
variable used within a generic class. Type variables can be invariant,
|
|
covariant, or contravariant. The concept of variance is an advanced detail
|
|
of type theory that is not well understood by most Python developers, yet
|
|
they are immediately confronted with the concept today when defining their
|
|
first generic class. This PEP largely eliminates the need for most developers
|
|
to understand the concept of variance when defining generic classes.
|
|
|
|
When more than one type parameter is used with a generic class or type alias,
|
|
the rules for type parameter ordering can be confusing. It is normally based on
|
|
the order in which they first appear within a class or type alias declaration
|
|
statement. However, this can be overridden in a class definition by
|
|
including a "Generic" or "Protocol" base class. For example, in the class
|
|
declaration ``class ClassA(Mapping[K, V])``, the type parameters are
|
|
ordered as ``K`` and then ``V``. However, in the class declaration
|
|
``class ClassB(Mapping[K, V], Generic[V, K])``, the type parameters are
|
|
ordered as ``V`` and then ``K``. This PEP proposes to make type parameter
|
|
ordering explicit in all cases.
|
|
|
|
The practice of sharing a type variable across multiple generic contexts
|
|
creates other problems today. Modern editors provide features like "find
|
|
all references" and "rename all references" that operate on symbols at the
|
|
semantic level. When a type parameter is shared among multiple generic
|
|
classes, functions, and type aliases, all references are semantically
|
|
equivalent.
|
|
|
|
Type variables defined within the global scope also need to be given a name
|
|
that starts with an underscore to indicate that the variable is private to
|
|
the module. Globally-defined type variables are also often given names to
|
|
indicate their variance, leading to cumbersome names like "_T_contra" and
|
|
"_KT_co". The current mechanisms for allocating type variables also requires
|
|
the developer to supply a redundant name in quotes (e.g. ``T = TypeVar("T")``).
|
|
This PEP eliminates the need for the redundant name and cumbersome
|
|
variable names.
|
|
|
|
Defining type parameters today requires importing the ``TypeVar`` and
|
|
``Generic`` symbols from the ``typing`` module. Over the past several releases
|
|
of Python, efforts have been made to eliminate the need to import ``typing``
|
|
symbols for common use cases, and the PEP furthers this goal.
|
|
|
|
|
|
Summary Examples
|
|
================
|
|
|
|
Defining a generic class prior to this PEP looks something like this.
|
|
|
|
::
|
|
|
|
from typing import Generic, TypeVar
|
|
|
|
_T_co = TypeVar("_T_co", covariant=True, bound=str)
|
|
|
|
class ClassA(Generic[_T_co]):
|
|
...
|
|
|
|
|
|
With the new syntax, it looks like this.
|
|
|
|
::
|
|
|
|
class ClassA[T: str]:
|
|
...
|
|
|
|
|
|
Here is an example of a generic function today.
|
|
|
|
::
|
|
|
|
from typing import ParamSpec, TypeVar, Callable
|
|
|
|
_P = ParamSpec("_P")
|
|
_R = TypeVar("_R")
|
|
|
|
def func(cb: Callable[_P, _R], *args: _P.args, **kwargs: _P.kwargs) -> _R:
|
|
...
|
|
|
|
And the new syntax.
|
|
|
|
::
|
|
|
|
from typing import Callable
|
|
|
|
def func[**P, R](cb: Callable[P, R], *args: P.args, **kwargs: P.kwargs) -> R:
|
|
...
|
|
|
|
|
|
Here is an example of a generic type alias today.
|
|
|
|
::
|
|
|
|
from typing import TypeAlias
|
|
|
|
_T = TypeVar("_T")
|
|
|
|
ListOrSet: TypeAlias = list[_T] | set[_T]
|
|
|
|
|
|
And with the new syntax.
|
|
|
|
::
|
|
|
|
type ListOrSet[T] = list[T] | set[T]
|
|
|
|
|
|
Specification
|
|
=============
|
|
|
|
Type Parameter Declarations
|
|
---------------------------
|
|
|
|
We propose to add new syntax for declaring type parameters for generic
|
|
classes, functions, and type aliases. The syntax adds support for
|
|
a comma-delimited list of type parameters in square brackets after
|
|
the name of the class, function, or type alias.
|
|
|
|
Simple (non-variadic) type variables are declared with an unadorned name.
|
|
Variadic type variables are preceded by ``*``. Parameter specifications are
|
|
preceded by ``**``.
|
|
|
|
::
|
|
|
|
# This generic class is parameterized by a TypeVar T, a
|
|
# TypeVarTuple Ts, and a ParamSpec P.
|
|
class ChildClass[T, *Ts, **P]: ...
|
|
|
|
There is no need to include ``Generic`` as a base class. Its inclusion as
|
|
a base class is implied by the presence of type parameters, and it will
|
|
automatically be included in the ``__mro__`` and ``__orig_bases`` attributes
|
|
for the class. The explicit use of a ``Generic`` base class will result in a
|
|
runtime error.
|
|
|
|
::
|
|
|
|
class ClassA[T](Generic[T]): ... # Runtime error
|
|
|
|
|
|
A ``Protocol`` base class with type arguments will not generate a runtime
|
|
error, but type checkers should generate an error in this case because
|
|
the use of type arguments is not needed, and the order of type parameters
|
|
for the class are no longer dictated by their order in the ``Protocol``
|
|
base class.
|
|
|
|
::
|
|
|
|
class ClassA[S, T](Protocol): ... # OK
|
|
|
|
class ClassB[S, T](Protocol[S, T]): ... # Recommended type checker error
|
|
|
|
|
|
Type parameter names within a generic class, function, or type alias must
|
|
be unique. Type parameters for a generic function cannot overlap the name
|
|
of a function parameter. A duplicate name generates a syntax error at compile
|
|
time.
|
|
|
|
::
|
|
|
|
class ClassA[T, *T]: ... # Syntax Error
|
|
|
|
def func1[T, **T](): ... # Syntax Error
|
|
|
|
def func2[T](T): ... # Syntax Error
|
|
|
|
|
|
Class type parameter names are not mangled if they begin with a double
|
|
underscore. Mangling would not make sense because type parameters, unlike other
|
|
class-scoped variables, cannot be accessed through the class dictionary, and
|
|
the notion of a "private" type parameter doesn't make sense. Other class-scoped
|
|
variables are mangled if they begin with a double underscore, so the mangled
|
|
name is used to determine whether there is a name collision with type parameters.
|
|
|
|
::
|
|
|
|
class ClassA[__T, _ClassA__S]:
|
|
__T = 0 # OK
|
|
__S = 0 # Syntax Error (because mangled name is _ClassA__S)
|
|
|
|
|
|
Type Parameter Scopes
|
|
---------------------
|
|
|
|
A type parameter declared as part of a generic class is valid within the
|
|
class body and inner scopes contained therein. Type parameters are also
|
|
accessible when evaluating the argument list (base classes and any keyword
|
|
arguments) that comprise the class definition. This allows base classes
|
|
to be parameterized by these type parameters. Type parameters are not
|
|
accessible outside of the class body, including in any class decorators.
|
|
|
|
::
|
|
|
|
class ClassA[T](BaseClass[T], param = Foo[T]): ... # OK
|
|
|
|
print(T) # Runtime error: 'T' is not defined
|
|
|
|
@dec(Foo[T]) # Runtime error: 'T' is not defined
|
|
class ClassA[T]: ...
|
|
|
|
A type parameter declared as part of a generic function is valid within
|
|
the function body and any scopes contained therein. It is also valid within
|
|
parameter and return type annotations. Default argument values for function
|
|
parameters are evaluated outside of this scope, so type parameters are
|
|
not accessible in default value expressions. Likewise, type parameters are not
|
|
in scope for function decorators.
|
|
|
|
::
|
|
|
|
def func1[T](a: T) -> T: ... # OK
|
|
|
|
print(T) # Runtime error: 'T' is not defined
|
|
|
|
def func2[T](a = list[T]): ... # Runtime error: 'T' is not defined
|
|
|
|
@dec(list[T]) # Runtime error: 'T' is not defined
|
|
def func3[T](): ...
|
|
|
|
|
|
Upper Bound Specification
|
|
-------------------------
|
|
|
|
For a non-variadic type parameter, an "upper bound" type can be specified
|
|
through the use of a type annotation expression. If an upper bound is
|
|
not specified, the upper bound is assumed to be ``object``.
|
|
|
|
::
|
|
|
|
class ClassA[T: str]: ...
|
|
|
|
The specified upper bound type must use an expression form that is allowed in
|
|
type annotations. More complex expression forms should be flagged
|
|
as an error by a type checker. Quoted forward declarations are allowed.
|
|
|
|
The specified upper bound type must be concrete. An attempt to use a generic
|
|
type should be flagged as an error by a type checker.
|
|
|
|
::
|
|
|
|
class ClassA[T: dict[str, int]]: ... # OK
|
|
|
|
class ClassB[T: "ForwardDeclaration"]: ... # OK
|
|
|
|
class ClassC[T: dict[str, V]]: ... # Type checker error: generic type
|
|
|
|
class ClassD[T: [str, int]]: ... # Type checker error: illegal expression form
|
|
|
|
|
|
Constrained Type Specification
|
|
------------------------------
|
|
|
|
For a non-variadic type parameter, a set of two or more "constrained types"
|
|
can be specified through the use of a literal tuple expression that contains
|
|
two or more types.
|
|
|
|
::
|
|
|
|
class ClassA[AnyStr: (str, bytes)]: ... # OK
|
|
|
|
class ClassB[T: ("ForwardDeclaration", bytes)]: ... # OK
|
|
|
|
class ClassC[T: ()]: ... # Type checker error: two or more types required
|
|
|
|
class ClassD[T: (str, )]: ... # Type checker error: two or more types required
|
|
|
|
t1 = (bytes, str)
|
|
class ClassE[T: t1]: ... # Type checker error: literal tuple expression required
|
|
|
|
|
|
If the specified type is not a tuple expression or the tuple expression includes
|
|
complex expression forms that are not allowed in a type annotation, a type
|
|
checker should generate an error. Quoted forward declarations are allowed.
|
|
|
|
::
|
|
|
|
class ClassF[T: (3, bytes)]: ... # Type checker error: invalid expression form
|
|
|
|
|
|
The specified constrained types must be concrete. An attempt to use a generic
|
|
type should be flagged as an error by a type checker. This is consistent with
|
|
the existing rules enforced by type checkers for a ``TypeVar`` constructor call.
|
|
|
|
::
|
|
|
|
class ClassG[T: (list[S], str)]: ... # Type checker error: generic type
|
|
|
|
|
|
|
|
Generic Type Alias
|
|
------------------
|
|
|
|
We propose to introduce a new statement for declaring type aliases. Similar
|
|
to ``class`` and ``def`` statements, a ``type`` statement defines a scope
|
|
for type parameters.
|
|
|
|
::
|
|
|
|
# A non-generic type alias
|
|
type IntOrStr = int | str
|
|
|
|
# A generic type alias
|
|
type ListOrSet[T] = list[T] | set[T]
|
|
|
|
|
|
Type aliases can refer to themselves without the use of quotes.
|
|
|
|
::
|
|
|
|
# A type alias that refers to a forward-declared type
|
|
type AnimalOrVegetable = Animal | "Vegetable"
|
|
|
|
# A generic self-referential type alias
|
|
type RecursiveList[T] = T | list[RecursiveList[T]]
|
|
|
|
|
|
The ``type`` keyword is a new soft keyword. It is interpreted as a keyword
|
|
only in this part of the grammar. In all other locations, it is assumed to
|
|
be an identifier name.
|
|
|
|
Type parameters declared as part of a generic type alias are valid only
|
|
when evaluating the right-hand side of the type alias.
|
|
|
|
As with ``typing.TypeAlias``, type checkers should restrict the right-hand
|
|
expression to expression forms that are allowed within type annotations.
|
|
The use of more complex expression forms (call expressions, ternary operators,
|
|
arithmetic operators, comparison operators, etc.) should be flagged as an
|
|
error.
|
|
|
|
Type alias expressions are not allowed to use traditional type variables.
|
|
Type checkers should generate an error in this case.
|
|
|
|
::
|
|
|
|
T = TypeVar("T")
|
|
type MyList = list[T] # Type checker error: traditional type variable usage
|
|
|
|
|
|
We propose to deprecate the existing ``typing.TypeAlias`` introduced in
|
|
:pep:`613`. The new syntax eliminates its need entirely.
|
|
|
|
|
|
Runtime Type Alias Class
|
|
------------------------
|
|
|
|
At runtime, a ``type`` statement will generate an instance of
|
|
``typing.TypeAliasType``. This class represents the type. Its attributes
|
|
include:
|
|
|
|
* ``__name__`` is a str representing the name of the type alias
|
|
* ``__parameters__`` is a tuple of ``TypeVar``, ``TypeVarTuple``, or
|
|
``ParamSpec`` objects that parameterize the type alias if it is generic
|
|
* ``__value__`` is the evaluated value of the type alias
|
|
|
|
The ``__value__`` attribute initially has a value of ``None`` while the type
|
|
alias expression is evaluated. It is then updated after a successful evaluation.
|
|
This allows for self-referential type aliases.
|
|
|
|
|
|
Variance Inference
|
|
------------------
|
|
|
|
We propose to eliminate the need for variance to be specified for type
|
|
parameters. Instead, type checkers will infer the variance of type parameters
|
|
based on their usage within a class. Type parameters can be invariant,
|
|
covariant, or contravariant depending on how they are used.
|
|
|
|
Python type checkers already include the ability to determine the variance of
|
|
type parameters for the purpose of validating variance within a generic
|
|
protocol class. This capability can be used for all classes (whether or not
|
|
they are protocols) to calculate the variance of each type parameter. This
|
|
eliminates the need for most developers to understand the concept of variance.
|
|
It also eliminates the need to introduce a dedicated syntax for specifying
|
|
variance.
|
|
|
|
The algorithm for computing the variance of a type parameter is as follows.
|
|
|
|
For each type parameter in a generic class:
|
|
|
|
1. If the type parameter is variadic (``TypeVarTuple``) or a parameter
|
|
specification (``ParamSpec``), it is always considered invariant. No further
|
|
inference is needed.
|
|
|
|
2. If the type parameter comes from a traditional ``TypeVar`` declaration and
|
|
is not specified as ``autovariance`` (see below), its variance is specified
|
|
by the ``TypeVar`` constructor call. No further inference is needed.
|
|
|
|
3. Create two specialized versions of the class. We'll refer to these as
|
|
``upper`` and ``lower`` specializations. In both of these specializations,
|
|
replace all type parameters other than the one being inferred by a dummy type
|
|
instance. In the ``upper`` specialized class, specialize the target type
|
|
parameter with an ``object`` instance. In the ``lower`` specialized class,
|
|
specialize the target type parameter with itself. This specialization
|
|
ignores the type parameter's upper bound or constraints.
|
|
|
|
4. Determine whether ``lower`` can be assigned to ``upper`` using normal type
|
|
compatibility rules. If so, the target type parameter is covariant. If not,
|
|
determine whether ``upper`` can be assigned to ``lower``. If so, the target
|
|
type parameter is contravariant. If neither of these combinations are
|
|
assignable, the target type parameter is invariant.
|
|
|
|
Here is an example.
|
|
|
|
::
|
|
|
|
class ClassA[T1, T2, T3](list[T1]):
|
|
def method1(self, a: T2) -> None:
|
|
...
|
|
|
|
def method2(self) -> T3:
|
|
...
|
|
|
|
To determine the variance of ``T1``, we specialize ``ClassA`` as follows:
|
|
|
|
::
|
|
|
|
upper = ClassA[object, Dummy, Dummy]
|
|
lower = ClassA[T1, Dummy, Dummy]
|
|
|
|
We find that ``upper`` is not assignable to ``lower`` nor is ``lower``
|
|
assignable to ``upper`` using standard type compatibility checks, so we
|
|
can conclude that ``T1`` is invariant.
|
|
|
|
To determine the variance of ``T2``, we specialize ``ClassA`` as follows:
|
|
|
|
::
|
|
|
|
upper = ClassA[Dummy, object, Dummy]
|
|
lower = ClassA[Dummy, T2, Dummy]
|
|
|
|
Since ``upper`` is assignable to ``lower``, ``T2`` is contravariant.
|
|
|
|
To determine the variance of ``T3``, we specialize ``ClassA`` as follows:
|
|
|
|
::
|
|
|
|
upper = ClassA[Dummy, Dummy, object]
|
|
lower = ClassA[Dummy, Dummy, T3]
|
|
|
|
Since ``lower`` is assignable to ``upper``, ``T3`` is covariant.
|
|
|
|
|
|
Auto Variance For TypeVar
|
|
-------------------------
|
|
|
|
The existing ``TypeVar`` class constructor accepts keyword parameters named
|
|
``covariant`` and ``contravariant``. If both of these are ``False``, the
|
|
type variable is assumed to be invariant. We propose to add another keyword
|
|
parameter named ``autovariance``. A corresponding instance variable
|
|
``__autovariance__`` can be accessed at runtime to determine whether the
|
|
variance is inferred. Type variables that are implicitly allocated using the
|
|
new syntax will always have ``__autovariance__`` set to ``True``.
|
|
|
|
A generic class that uses the traditional syntax may include combinations of
|
|
type variables with explicit and inferred variance.
|
|
|
|
::
|
|
|
|
T1 = TypeVar("T1", autovariance=True) # Inferred variance
|
|
T2 = TypeVar("T2") # Invariant
|
|
T3 = TypeVar("T3", covariant=True) # Covariant
|
|
|
|
# A type checker should infer the variance for T1 but use the
|
|
# specified variance for T2 and T3.
|
|
class ClassA(Generic[T1, T2, T3]): ...
|
|
|
|
|
|
Compatibility with Traditional TypeVars
|
|
---------------------------------------
|
|
|
|
The existing mechanism for allocating ``TypeVar``, ``TypeVarTuple``, and
|
|
``ParamSpec`` is retained for backward compatibility. However, these
|
|
"traditional" type variables should not be combined with type parameters
|
|
allocated using the new syntax. Such a combination should be flagged as
|
|
an error by type checkers. This is necessary because the type parameter
|
|
order is ambiguous.
|
|
|
|
It is OK to combine traditional type variables with new-style type parameters
|
|
if the class, function, or type alias does not use the new syntax. The
|
|
new-style type parameters must come from an outer scope in this case.
|
|
|
|
::
|
|
|
|
K = TypeVar("K")
|
|
|
|
class ClassA[V](dict[K, V]): ... # Type checker error
|
|
|
|
class ClassB[K, V](dict[K, V]): ... # OK
|
|
|
|
class ClassC[V]:
|
|
# The use of K and V for "method1" is OK because it uses the
|
|
# "traditional" generic function mechanism where type parameters
|
|
# are implicit. In this case V comes from an outer scope (ClassC)
|
|
# and K is introduced implicitly as a type parameter for "method1".
|
|
def method1(self, a: V, b: K) -> V | K: ...
|
|
|
|
# The use of M and K are not allowed for "method2". A type checker
|
|
# should generate an error in this case because this method uses the
|
|
# new syntax for type parameters, and all type parameters associated
|
|
# with the method must be explicitly declared. In this case, ``K``
|
|
# is not declared by "method2", nor is it supplied defined an outer
|
|
# scope.
|
|
def method2[M](self, a: M, b: K) -> M | K: ...
|
|
|
|
|
|
Runtime Implementation
|
|
======================
|
|
|
|
Grammar Changes
|
|
---------------
|
|
|
|
This PEP introduces a new soft keyword ``type``. It modifies the grammar
|
|
in the following ways:
|
|
|
|
1. Addition of optional type parameter clause in ``class`` and ``def`` statements.
|
|
|
|
::
|
|
|
|
type_params: '[' t=type_param_seq ']'
|
|
|
|
type_param_seq: a[asdl_typeparam_seq*]=','.type_param+ [',']
|
|
|
|
type_param:
|
|
| a=NAME b=[type_param_bound]
|
|
| '*' a=NAME
|
|
| '**' a=NAME
|
|
|
|
type_param_bound: ":" e=expression
|
|
|
|
|
|
2. Addition of new ``type`` statement for defining type aliases.
|
|
|
|
::
|
|
|
|
type_alias[stmt_ty]:
|
|
| "type" n=NAME t=[type_params] '=' b=expression {
|
|
CHECK_VERSION(stmt_ty, 12, "Type statement is", _PyAST_TypeAlias(n->v.Name.id, t, b, EXTRA)) }
|
|
|
|
|
|
AST Changes
|
|
-----------
|
|
|
|
This PEP introduces a new AST node type called ``TypeAlias``.
|
|
|
|
::
|
|
|
|
TypeAlias(identifier name, typeparam* typeparams, expr value)
|
|
|
|
It also adds an AST node that represents a type parameter.
|
|
|
|
::
|
|
|
|
typeparam = TypeVar(identifier name, expr? bound)
|
|
| ParamSpec(identifier name)
|
|
| TypeVarTuple(identifier name)
|
|
|
|
It also modifies existing AST nodes ``FunctionDef``, ``AsyncFunctionDef`` and
|
|
``ClassDef`` to include an additional optional attribute called ``typeparam*``
|
|
that includes a list of type parameters associated with the function or class.
|
|
|
|
|
|
Compiler Changes
|
|
----------------
|
|
|
|
The compiler maintains a list of "active type variables" as it recursively
|
|
generates byte codes for the program. Consider the following example.
|
|
|
|
::
|
|
|
|
class Outer[K, V]:
|
|
# Active type variables are K and V
|
|
|
|
class Inner[T]:
|
|
# Active type variables are K, V, and T
|
|
|
|
def method[M](self, a: M) -> M:
|
|
# Active type variables are K, V, T, and M
|
|
...
|
|
|
|
An active type variable symbol cannot be used for other purposes within
|
|
these scopes. This includes local parameters, local variables, variables
|
|
bound from other scopes (nonlocal or global), or other type parameters. An
|
|
attempt to reuse a type variable name in one of these manners results in
|
|
a syntax error.
|
|
|
|
::
|
|
|
|
class ClassA[K, V]:
|
|
class Inner[K]: # Syntax error: K already in use as type variable
|
|
...
|
|
|
|
class ClassB[K, V]:
|
|
def method(self, K): # Syntax error: K already in use as type variable
|
|
...
|
|
|
|
class ClassC[T, T]: # Syntax error: T already in use as type variable
|
|
...
|
|
|
|
def func1[T]():
|
|
...
|
|
|
|
|
|
A type variable is considered "active" when compiling the arguments for
|
|
a class declaration, the type annotations for a function declaration, and
|
|
the right-hand expression in a type alias declaration. Type variable are
|
|
not considered "active" when compiling the default argument expressions for
|
|
a function declaration or decorator expressions for classes or functions.
|
|
|
|
::
|
|
|
|
T = list
|
|
|
|
@decorator(T) # T in decorator refers to outer variable
|
|
class ClassA[T](Base[T], metaclass=Meta[T]) # T refers to type variable
|
|
...
|
|
|
|
@decorator(T) # T in decorator refers to outer variable
|
|
def func1[T](a: list[T]) -> T: # T refers to type variable
|
|
...
|
|
|
|
def func2[T](a = T): # T in default refers to outer variable
|
|
...
|
|
|
|
|
|
When a ``class``, ``def``, or ``type`` statement includes one or more type
|
|
parameters, the compiler emits byte codes to construct the corresponding
|
|
``typing.TypeVar``, ``typing.TypeVarTuple``, or ``typing.ParamSpec`` instances.
|
|
It then builds a new tuple that includes all active type variables and stores
|
|
this new tuple in a local variable. Active type variables include all type
|
|
parameters declared by outer ``class`` and ``def`` scopes plus those declared
|
|
by the ``class``, ``def``, or ``type`` statement itself. (In the reference
|
|
implementation, the local variable happens to have the name
|
|
``__type_variables__``, but this is an implementation detail. Other Python
|
|
compilers or future versions of the CPython compiler may choose a different
|
|
name or an entirely anonymous local variable slot for this purpose.)
|
|
|
|
When a type variable is referenced, the compiler generates opcodes that
|
|
load the active type variable tuple from either the local variable or (if
|
|
there are no local type variables) through the use of a new opcode called
|
|
``LOAD_TYPEVARS`` that loads the tuple of active type variables from the
|
|
current function object. It then emits opcodes to index into this tuple to
|
|
fetch the desired type variable.
|
|
|
|
When a new function is created, the "active type variables" tuple is copied
|
|
to the C struct field ``func_typevars`` of the function object, making the type
|
|
variables from outer scopes available to inner scopes of the function or class.
|
|
|
|
A new read-only attribute called ``__type_variables__`` is available on class,
|
|
function, and type alias objects. This attribute is a tuple of the active
|
|
type variables that are visible within the scope of that class, function,
|
|
or type alias. This attribute is used for runtime evaluation of stringified
|
|
(forward referenced) type annotations that include references to type
|
|
parameters. Functions like ``typing.get_type_hints`` can use this attribute
|
|
to populate the ``locals`` dictionary with values for type parameters that
|
|
are in scope when calling ``eval`` to evaluate the stringified expression.
|
|
|
|
|
|
Library Changes
|
|
---------------
|
|
|
|
Several classes in the ``typing`` module that are currently implemented in
|
|
Python must be reimplemented in C. This includes: ``TypeVar``,
|
|
``TypeVarTuple``, ``ParamSpec``, and ``Generic``. The new class
|
|
``TypeAliasType`` (described above) also must be implemented in C.
|
|
|
|
The ``typing.get_type_hints`` must be updated to use the new
|
|
``__type_variables__`` attribute.
|
|
|
|
|
|
Reference Implementation
|
|
========================
|
|
|
|
This proposal is prototyped in the CPython code base in
|
|
`this fork <https://github.com/erictraut/cpython/tree/type_param_syntax2>`_.
|
|
|
|
The Pyright type checker supports the behavior described in this PEP.
|
|
|
|
|
|
Rejected Ideas
|
|
==============
|
|
|
|
Prefix Clause
|
|
-------------
|
|
We explored various syntactic options for specifying type parameters that
|
|
preceded ``def`` and ``class`` statements. One such variant we considered
|
|
used a ``using`` clause as follows:
|
|
|
|
::
|
|
|
|
using S, T
|
|
class ClassA: ...
|
|
|
|
This option was rejected because the scoping rules for the type parameters
|
|
were less clear. Also, this syntax did not interact well with class and
|
|
function decorators, which are common in Python. Only one other popular
|
|
programming language, C++, uses this approach.
|
|
|
|
We likewise considered prefix forms that looked like decorators (e.g.,
|
|
``@using(S, T)``). This idea was rejected because such forms would be confused
|
|
with regular decorators, and they would not compose well with existing
|
|
decorators. Furthermore, decorators are logically executed after the statement
|
|
they are decorating, so it would be confusing for them to introduce symbols
|
|
(type parameters) that are visible within the "decorated" statement, which is
|
|
logically executed before the decorator itself.
|
|
|
|
|
|
Angle Brackets
|
|
--------------
|
|
Many languages that support generics make use of angle brackets. (Refer to
|
|
the table at the end of Appendix A for a summary.) We explored the use of
|
|
angle brackets for type parameter declarations in Python, but we ultimately
|
|
rejected it for two reasons. First, angle brackets are not considered
|
|
"paired" by the Python scanner, so end-of-line characters between a ``<``
|
|
and ``>`` token are retained. That means any line breaks within a list of
|
|
type parameters would require the use of unsightly and cumbersome ``\`` escape
|
|
sequences. Second, Python has already established the use of square brackets
|
|
for explicit specialization of a generic type (e.g., ``list[int]``). We
|
|
concluded that it would be inconsistent and confusing to use angle brackets
|
|
for generic declarations but square brackets for explicit specialization. All
|
|
other languages that we surveyed were consistent in this regard.
|
|
|
|
|
|
Bounds Syntax
|
|
-------------
|
|
We explored various syntactic options for specifying the bounds and constraints
|
|
for a type variable. We considered, but ultimately rejected, the use
|
|
of a ``<:`` token like in Scala, the use of an ``extends`` or ``with``
|
|
keyword like in various other languages, and the use of a function call
|
|
syntax similar to today's ``typing.TypeVar`` constructor. The simple colon
|
|
syntax is consistent with many other programming languages (see Appendix A),
|
|
and it was heavily preferred by a cross section of Python developers who
|
|
were surveyed.
|
|
|
|
|
|
Explicit Variance
|
|
-----------------
|
|
We considered adding syntax for specifying whether a type parameter is intended
|
|
to be invariant, covariant, or contravariant. The ``typing.TypeVar`` mechanism
|
|
in Python requires this. A few other languages including Scala and C#
|
|
also require developers to specify the variance. We rejected this idea because
|
|
variance can generally be inferred, and most modern programming languages
|
|
do infer variance based on usage. Variance is an advanced topic that
|
|
many developers find confusing, so we want to eliminate the need to
|
|
understand this concept for most Python developers.
|
|
|
|
|
|
Name Mangling
|
|
-------------
|
|
When considering implementation options, we considered a "name mangling"
|
|
approach where each type parameter was given a unique "mangled" name by the
|
|
compiler. This mangled name would be based on the qualified name of the
|
|
generic class, function or type alias it was associated with. This approach
|
|
was rejected because qualified names are not necessarily unique, which means
|
|
the mangled name would need to be based on some other randomized value.
|
|
Furthermore, this approach is not compatible with techniques used for
|
|
evaluating quoted (forward referenced) type annotations.
|
|
|
|
|
|
Lambda Lifting
|
|
--------------
|
|
When considering implementation options, we considered introducing a new
|
|
scope and executing the ``class``, ``def``, or ``type`` statement within
|
|
a lambda -- a technique that is sometimes referred to as "lambda lifting".
|
|
We ultimately rejected this idea because it did not work well for statements
|
|
within a class body (because class-scoped symbols cannot be accessed by
|
|
inner scopes). It also introduced many odd behaviors for scopes that were
|
|
further nested within the lambda.
|
|
|
|
|
|
Appendix A: Survey of Type Parameter Syntax
|
|
===========================================
|
|
|
|
Support for generic types is found in many programming languages. In this
|
|
section, we provide a survey of the options used by other popular programming
|
|
languages. This is relevant because familiarity with other languages will
|
|
make it easier for Python developers to understand this concept. We provide
|
|
additional details here (for example, default type argument support) that
|
|
may be useful when considering future extensions to the Python type system.
|
|
|
|
|
|
C++
|
|
---
|
|
|
|
C++ uses angle brackets in combination with keywords ``template`` and
|
|
``typename`` to declare type parameters. It uses angle brackets for
|
|
specialization.
|
|
|
|
C++20 introduced the notion of generalized constraints, which can act
|
|
like protocols in Python. A collection of constraints can be defined in
|
|
a named entity called a ``concept``.
|
|
|
|
Variance is not explicitly specified, but constraints can enforce variance.
|
|
|
|
A default type argument can be specified using the ``=`` operator.
|
|
|
|
.. code-block:: c++
|
|
|
|
// Generic class
|
|
template <typename>
|
|
class ClassA
|
|
{
|
|
// Constraints are supported through compile-time assertions.
|
|
static_assert(std::is_base_of<BaseClass, T>::value);
|
|
|
|
public:
|
|
Container<T> t;
|
|
};
|
|
|
|
// Generic function with default type argument
|
|
template <typename S = int>
|
|
S func1(ClassA<S> a, S b) {};
|
|
|
|
// C++20 introduced a more generalized notion of "constraints"
|
|
// and "concepts", which are named constraints.
|
|
|
|
// A sample concept
|
|
template<typename T>
|
|
concept Hashable = requires(T a)
|
|
{
|
|
{ std::hash<T>{}(a) } -> std::convertible_to<std::size_t>;
|
|
};
|
|
|
|
// Use of a concept in a template
|
|
template<Hashable T>
|
|
void func2(T value) {}
|
|
|
|
// Alternative use of concept
|
|
template<typename T> requires Hashable<T>
|
|
void func3(T value) {}
|
|
|
|
// Alternative use of concept
|
|
template<typename T>
|
|
void func3(T value) requires Hashable<T> {}
|
|
|
|
|
|
Java
|
|
----
|
|
|
|
Java uses angle brackets to declare type parameters and for specialization.
|
|
By default, type parameters are invariant.
|
|
The ``extends`` keyword is used to specify an upper bound. The ``super`` keyword
|
|
is used to specify a contravariant bound.
|
|
|
|
Java uses use-site variance. The compiler places limits on which methods and
|
|
members can be accessed based on the use of a generic type. Variance is
|
|
not specified explicitly.
|
|
|
|
Java provides no way to specify a default type argument.
|
|
|
|
.. code-block:: java
|
|
|
|
// Generic class
|
|
public class ClassA<T> {
|
|
public Container<T> t;
|
|
|
|
// Generic method
|
|
public <S extends Number> void method1(S value) { }
|
|
|
|
// Use site variance
|
|
public void method1(ClassA<? super Integer> value) { }
|
|
}
|
|
|
|
|
|
C#
|
|
--
|
|
|
|
C# uses angle brackets to declare type parameters and for specialization.
|
|
The ``where`` keyword and a colon is used to specify the bound for a type
|
|
parameter.
|
|
|
|
C# uses declaration-site variance using the keywords ``in`` and ``out`` for
|
|
contravariance and covariance, respectively. By default, type parameters are
|
|
invariant.
|
|
|
|
C# provides no way to specify a default type argument.
|
|
|
|
.. code-block:: c#
|
|
|
|
// Generic class with bounds on type parameters
|
|
public class ClassA<S, T>
|
|
where T : SomeClass1
|
|
where S : SomeClass2
|
|
{
|
|
// Generic method
|
|
public void MyMethod<U>(U value) where U : SomeClass3 { }
|
|
}
|
|
|
|
// Contravariant and covariant type parameters
|
|
public class ClassB<in S, out T>
|
|
{
|
|
public T MyMethod(S value) { }
|
|
}
|
|
|
|
|
|
|
|
TypeScript
|
|
----------
|
|
|
|
TypeScript uses angle brackets to declare type parameters and for
|
|
specialization. The ``extends`` keyword is used to specify a bound. It can be
|
|
combined with other type operators such as ``keyof``.
|
|
|
|
TypeScript uses declaration-site variance. Variance is inferred from
|
|
usage, not specified explicitly. TypeScript 4.7 introduced the ability
|
|
to specify variance using ``in`` and ``out`` keywords. This was added to handle
|
|
extremely complex types where inference of variance was expensive,
|
|
yet the maintainers state that is useful for increasing readability.
|
|
|
|
A default type argument can be specified using the ``=`` operator.
|
|
|
|
TypeScript supports the ``type`` keyword to declare a type alias, and this
|
|
syntax supports generics.
|
|
|
|
.. code-block:: typescript
|
|
|
|
// Generic interface
|
|
interface InterfaceA<S, T extends SomeInterface1> {
|
|
val1: S;
|
|
val2: T;
|
|
|
|
method1<U extends SomeInterface2>(val: U): S
|
|
}
|
|
|
|
// Generic function
|
|
function func1<T, K extends keyof T>(ojb: T, key: K) { }
|
|
|
|
// Contravariant and covariant type parameters (TypeScript 4.7)
|
|
interface InterfaceB<in S, out T> { }
|
|
|
|
// Type parameter with default
|
|
interface InterfaceC<T = SomeInterface3> { }
|
|
|
|
// Generic type alias
|
|
type MyType<T extends SomeInterface4> = Array<T>
|
|
|
|
|
|
Scala
|
|
-----
|
|
|
|
In Scala, square brackets are used to declare type parameters. Square
|
|
brackets are also used for specialization. The ``<:`` and ``>:`` operators
|
|
are used to specify upper and lower bounds, respectively.
|
|
|
|
Scala uses use-site variance but also allows declaration-site variance
|
|
specification. It uses a ``+`` or ``-`` prefix operator for covariance and
|
|
contravariance, respectively.
|
|
|
|
Scala provides no way to specify a default type argument.
|
|
|
|
It does support higher-kinded types (type parameters that accept type
|
|
type parameters).
|
|
|
|
.. code-block:: scala
|
|
|
|
|
|
// Generic class; type parameter has upper bound
|
|
class ClassA[A <: SomeClass1]
|
|
{
|
|
// Generic method; type parameter has lower bound
|
|
def method1[B >: A](val: B) ...
|
|
}
|
|
|
|
// Use of an upper and lower bound with the same type parameter
|
|
class ClassB[A >: SomeClass1 <: SomeClass2] { }
|
|
|
|
// Contravariant and covariant type parameters
|
|
class ClassC[+A, -B] { }
|
|
|
|
// Higher-kinded type
|
|
trait Collection[T[_]]
|
|
{
|
|
def method1[A](a: A): T[A]
|
|
def method2[B](b: T[B]): B
|
|
}
|
|
|
|
// Generic type alias
|
|
type MyType[T <: Int] = Container[T]
|
|
|
|
|
|
Swift
|
|
-----
|
|
|
|
Swift uses angle brackets to declare type parameters and for specialization.
|
|
The upper bound of a type parameter is specified using a colon.
|
|
|
|
Swift doesn't support generic variance; all type parameters are invariant.
|
|
|
|
Swift provides no way to specify a default type argument.
|
|
|
|
.. code-block:: swift
|
|
|
|
// Generic class
|
|
class ClassA<T> {
|
|
// Generic method
|
|
func method1<X>(val: T) -> X { }
|
|
}
|
|
|
|
// Type parameter with upper bound constraint
|
|
class ClassB<T: SomeClass1> {}
|
|
|
|
// Generic type alias
|
|
typealias MyType<A> = Container<A>
|
|
|
|
|
|
Rust
|
|
----
|
|
|
|
Rust uses angle brackets to declare type parameters and for specialization.
|
|
The upper bound of a type parameter is specified using a colon. Alternatively
|
|
a ``where`` clause can specify various constraints.
|
|
|
|
Rust does not have traditional object oriented inheritance or variance.
|
|
Subtyping in Rust is very restricted and occurs only due to variance with
|
|
respect to lifetimes.
|
|
|
|
A default type argument can be specified using the ``=`` operator.
|
|
|
|
.. code-block:: rust
|
|
|
|
// Generic class
|
|
struct StructA<T> { // T's lifetime is inferred as covariant
|
|
x: T
|
|
}
|
|
|
|
fn f<'a>(
|
|
mut short_lifetime: StructA<&'a i32>,
|
|
mut long_lifetime: StructA<&'static i32>,
|
|
) {
|
|
long_lifetime = short_lifetime;
|
|
// error: StructA<&'a i32> is not a subtype of StructA<&'static i32>
|
|
short_lifetime = long_lifetime;
|
|
// valid: StructA<&'static i32> is a subtype of StructA<&'a i32>
|
|
}
|
|
|
|
// Type parameter with bound
|
|
struct StructB<T: SomeTrait> {}
|
|
|
|
// Type parameter with additional constraints
|
|
struct StructC<T>
|
|
where
|
|
T: Iterator,
|
|
T::Item: Copy
|
|
{}
|
|
|
|
// Generic function
|
|
fn func1<T>(val: &[T]) -> T { }
|
|
|
|
// Generic type alias
|
|
type MyType<T> = StructC<T>;
|
|
|
|
|
|
Kotlin
|
|
------
|
|
|
|
Kotlin uses angle brackets to declare type parameters and for specialization.
|
|
By default, type parameters are invariant. The upper bound of a type is
|
|
specified using a colon.
|
|
Alternatively, a ``where`` clause can specify various constraints.
|
|
|
|
Kotlin supports declaration-site variance where variance of type parameters is
|
|
explicitly declared using ``in`` and ``out`` keywords. It also supports use-site
|
|
variance which limits which methods and members can be used.
|
|
|
|
Kotlin provides no way to specify a default type argument.
|
|
|
|
.. code-block:: kotlin
|
|
|
|
// Generic class
|
|
class ClassA<T>
|
|
|
|
// Type parameter with upper bound
|
|
class ClassB<T : SomeClass1>
|
|
|
|
// Contravariant and covariant type parameters
|
|
class ClassC<in S, out T>
|
|
|
|
// Generic function
|
|
fun <T> func1(): T {
|
|
|
|
// Use site variance
|
|
val covariantA: ClassA<out Number>
|
|
val contravariantA: ClassA<in Number>
|
|
}
|
|
|
|
// Generic type alias
|
|
typealias TypeAliasFoo<T> = ClassA<T>
|
|
|
|
|
|
Julia
|
|
-----
|
|
|
|
Julia uses curly braces to declare type parameters and for specialization.
|
|
The ``<:`` operator can be used within a ``where`` clause to declare
|
|
upper and lower bounds on a type.
|
|
|
|
.. code-block:: julia
|
|
|
|
// Generic struct; type parameter with upper and lower bounds
|
|
struct StructA{T} where Int <: T <: Number
|
|
x::T
|
|
end
|
|
|
|
// Generic function
|
|
function func1{T <: Real}(v::Container{T})
|
|
|
|
// Alternate form of generic function
|
|
function func2(v::Container{T} where T <: Real)
|
|
|
|
Dart
|
|
----
|
|
|
|
Dart uses angle brackets to declare type parameters and for specialization.
|
|
The upper bound of a type is specified using the ``extends`` keyword.
|
|
By default, type parameters are covariant.
|
|
|
|
Dart supports declaration-site variance, where variance of type parameters is
|
|
explicitly declared using ``in``, ``out`` and ``inout`` keywords.
|
|
It does not support use-site variance.
|
|
|
|
Dart provides no way to specify a default type argument.
|
|
|
|
.. code-block:: dart
|
|
|
|
// Generic class
|
|
class ClassA<T> { }
|
|
|
|
// Type parameter with upper bound
|
|
class ClassB<T extends SomeClass1> { }
|
|
|
|
// Contravariant and covariant type parameters
|
|
class ClassC<in S, out T> { }
|
|
|
|
// Generic function
|
|
T func1<T>() { }
|
|
|
|
// Generic type alias
|
|
typedef TypeDefFoo<T> = ClassA<T>;
|
|
|
|
Go
|
|
--
|
|
|
|
Go uses square brackets to declare type parameters and for specialization.
|
|
The upper bound of a type is specified after the name of the parameter, and
|
|
must always be specified. The keyword ``any`` is used for an unbound type parameter.
|
|
|
|
Go doesn't support variance; all type parameters are invariant.
|
|
|
|
Go provides no way to specify a default type argument.
|
|
|
|
Go does not support generic type aliases.
|
|
|
|
.. code-block:: go
|
|
|
|
// Generic type without a bound
|
|
type TypeA[T any] struct {
|
|
t T
|
|
}
|
|
|
|
// Type parameter with upper bound
|
|
type TypeB[T SomeType1] struct { }
|
|
|
|
// Generic function
|
|
func func1[T any]() { }
|
|
|
|
|
|
Summary
|
|
-------
|
|
|
|
+------------+----------+---------+--------+----------+-----------+-----------+
|
|
| | Decl | Upper | Lower | Default | Variance | Variance |
|
|
| | Syntax | Bound | Bound | Value | Site | |
|
|
+============+==========+=========+========+==========+===========+===========+
|
|
| C++ | template | n/a | n/a | = | n/a | n/a |
|
|
| | <> | | | | | |
|
|
+------------+----------+---------+--------+----------+-----------+-----------+
|
|
| Java | <> | extends | | | use | super, |
|
|
| | | | | | | extends |
|
|
+------------+----------+---------+--------+----------+-----------+-----------+
|
|
| C# | <> | where | | | decl | in, out |
|
|
+------------+----------+---------+--------+----------+-----------+-----------+
|
|
| TypeScript | <> | extends | | = | decl | inferred, |
|
|
| | | | | | | in, out |
|
|
+------------+----------+---------+--------+----------+-----------+-----------+
|
|
| Scala | [] | T <: X | T >: X | | use, decl | +, - |
|
|
+------------+----------+---------+--------+----------+-----------+-----------+
|
|
| Swift | <> | T: X | | | n/a | n/a |
|
|
+------------+----------+---------+--------+----------+-----------+-----------+
|
|
| Rust | <> | T: X, | | = | n/a | n/a |
|
|
| | | where | | | | |
|
|
+------------+----------+---------+--------+----------+-----------+-----------+
|
|
| Kotlin | <> | T: X, | | | use, decl | in, out |
|
|
| | | where | | | | |
|
|
+------------+----------+---------+--------+----------+-----------+-----------+
|
|
| Julia | {} | T <: X | X <: T | | n/a | n/a |
|
|
+------------+----------+---------+--------+----------+-----------+-----------+
|
|
| Dart | <> | extends | | | decl | in, out, |
|
|
| | | | | | | inout |
|
|
+------------+----------+---------+--------+----------+-----------+-----------+
|
|
| Go | [] | T X | | | n/a | n/a |
|
|
+------------+----------+---------+--------+----------+-----------+-----------+
|
|
| Python | [] | T: X | | | decl | inferred |
|
|
| (proposed) | | | | | | |
|
|
+------------+----------+---------+--------+----------+-----------+-----------+
|
|
|
|
|
|
Acknowledgements
|
|
================
|
|
|
|
Thanks to Sebastian Rittau for kick-starting the discussions that led to this
|
|
proposal, to Jukka Lehtosalo for proposing the syntax for type alias
|
|
statements and to Jelle Zijlstra, Daniel Moisset, and Guido van Rossum
|
|
for their valuable feedback and suggested improvements to the specification
|
|
and implementation.
|
|
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document is placed in the public domain or under the CC0-1.0-Universal
|
|
license, whichever is more permissive.
|