diff --git a/pep-0646.rst b/pep-0646.rst index cc51cc100..1d9996b54 100644 --- a/pep-0646.rst +++ b/pep-0646.rst @@ -1,4 +1,4 @@ -PEP: 0646 +PEP: 646 Title: Variadic Generics Author: Mark Mendoza , Matthew Rahtz , @@ -26,20 +26,6 @@ to catch shape-related bugs in code that uses these libraries. Motivation ========== -There are two main use-cases for variadic generics. [#hkt]_ - -The primary motivation is to enable typing of array shapes in numerical -computing libraries such as NumPy and TensorFlow. This is the use-case -much of the PEP will focus on. - -Additionally, variadic generics allow us to concisely specify the type -signature of ``map`` and ``zip``. - -We discuss each of these motivations below. - -Array Shapes -------------- - In the context of numerical computation with libraries such as NumPy and TensorFlow, the *shape* of arguments is often just as important as the argument *type*. For example, consider the following function which converts a @@ -87,56 +73,10 @@ and so on throughout their code) and for the authors of array libraries (who wou Variadic generics are necessary for a ``Array`` that is generic in an arbitrary number of axes to be cleanly defined as a single class. -``map`` and ``zip`` -------------------- - -PEP 612 [#pep-612]_ introduced ``ParamSpec``, enabling parameter types of one -callable to be forwarded to another callable. However, in many cases we actually -wish to *transform* parameter types before using them elsewhere in the -signature. - -Consider, for example, the signature of ``map`` for a particular choice of -function and iterables: - -:: - - def func(int, str) -> float: ... - iter1: List[int] - iter2: List[str] - - def map(func: Callable[[int, str], float], - iter1: Iterable[int], - iter2: Iterable[str]) -> Iterable[float]: ... - -Note that the number of iterables passed to ``map`` is dependent -on the supplied function, and that the types of those iterables -must correspond to the types of supplied function's arguments. - -A similar example is ``zip``: - -:: - - iter1: List[int] - iter2: List[str] - - def zip(iter1: Iterable[int], - iter2: Iterable[str]) -> Iterable[Tuple[int, str]]: ... - -Neither of these signatures can be specified in the general case using -existing typing mechanisms. The signature of ``zip``, for example, is -currently specified [#zip-sig]_ with a number of overloads. - Specification ============= -In order to support the above use-cases, we introduce: - -* ``TypeVarTuple``, serving as a placeholder not for a single type but - for an *arbitrary* number of types, and behaving like a number of - ``TypeVar`` instances packed in a ``Tuple``. -* A new use for the star operator: unpacking of each individual type - from a ``TypeVarTuple``. -* Two new type operators, ``Unpack`` and ``Map``. +In order to support the above use cases, we introduce ``TypeVarTuple``. This serves as a placeholder not for a single type but for an *arbitrary* number of types, and behaving like a number of ``TypeVar`` instances packed in a ``Tuple``. These are described in detail below. @@ -155,130 +95,293 @@ Type variable tuples are created with: Ts = TypeVarTuple('Ts') -A type variable tuple behaves in a similar way to a parameterized ``Tuple``. -For example, in a generic object instantiated with type parameters -``int`` and ``str``, ``Ts`` is equivalent to ``Tuple[int, str]``. +Type variable tuples behave like a number of individual type variables packed in a +``Tuple``. To understand this, consider the following example: + +:: + + Shape = TypeVarTuple('Shape') + + class Array(Generic[*Shape]): ... + + Height = NewType('Height', int) + Width = NewType('Width', int) + x: Array[Height, Width] = Array() + +The ``Shape`` type variable tuple here behaves like ``Tuple[T1, T2]``, +where ``T1`` and ``T2`` are type variables. To use these type variables +as type parameters of ``Array``, we must **unpack** the type variable tuple using +the star operator: ``*Shape``. The signature of ``Array`` then behaves +as if we had simply written ``class Array(Generic[T1, T2]): ...``. + +In contrast to ``Generic[T1, T2]``, however, ``Generic[*Shape]`` allows +us to parameterise the class with an *arbitrary* number of type parameters. +That is, in addition to being able to define rank-2 arrays such as +``Array[Height, Width]``, we could also define rank-3 arrays, rank-4 arrays, +and so on: + +:: + + Time = NewType('Time', int) + Batch = NewType('Batch', int) + y: Array[Batch, Height, Width] = Array() + z: Array[Time, Batch, Height, Width] = Array() Type variable tuples can be used anywhere a normal ``TypeVar`` can. -For example, in class definitions, function signatures, and variable annotations: +This includes class definitions, as shown above, as well as function +signatures and variable annotations: :: - Shape = TypeTupleVar('Shape') - - class Array(Generic[Shape]): - - def __init__(self, shape: Shape): - self.shape: Shape = shape - - def __abs__(self) -> Array[Shape]: ... - - def __add__(self, other: Array[Shape]) -> Array[Shape]: ... - - Height = NewType('Height', int) - Width = NewType('Width', int) - shape = (Height(480), Width(640)) - x: Array[Tuple[Height, Width]] = Array(shape) - x.shape # Inferred type is Tuple[Height, Width] - y = abs(x) # Array[Tuple[Height, Width]] - z = x + y # Array[Tuple[Height, Width]] - -Variance and ``bound``: Not (Yet) Supported -''''''''''''''''''''''''''''''''''''''''''' - -To keep this PEP minimal, ``TypeTupleVar`` does not yet support -the ``bound`` argument or specification of variance, as ``TypeVar`` -does. We leave the decision of how these arguments should be implemented -to a future PEP, when use-cases for variadic generics have been -explored more in practice. - -Unpacking: Star Operator -'''''''''''''''''''''''' - -Note that the fully-parameterised type of ``Array`` above is -rather verbose. Wouldn't it be easier if we could just write -``Array[Height, Width]``? - -To enable this, we introduce a new function for the star operator: -to 'unpack' type variable tuples. When unpacked, a type variable tuple -behaves as if its component types had been written -directly into the signature, rather than being wrapped in a ``Tuple``. - -Rewriting the ``Array`` class using an unpacked type variable -tuple, we can instead write: - -:: - - Shape = TypeTupleVar('Shape') - class Array(Generic[*Shape]): - def __init__(self, shape: Shape): - self.shape: Shape = shape + def __init__(self, shape: Tuple[*Shape]): + self._shape: Tuple[*Shape] = shape - def __add__(self, other: Array[*Shape]) -> Array[*Shape]: ... + def get_shape(self) -> Tuple[*Shape]: + return self._shape + + def __abs__(self) -> Array[*Shape]: ... + + def __add__(self, other: Array[*Shape]) -> Array[*Shape]) ... shape = (Height(480), Width(640)) x: Array[Height, Width] = Array(shape) - x.shape # Inferred type is Tuple[Height, Width] - z = x + x # Array[Height, Width] + y = abs(x) # Inferred type is Array[Height, Width] + z = x + x # ... is Array[Height, Width] -Unpacking: ``Unpack`` Operator -'''''''''''''''''''''''''''''' +Type Variable Tuples Must Always be Unpacked +'''''''''''''''''''''''''''''''''''''''''''' -Because the new use of the star operator requires a syntax change and is -therefore incompatible with previous versions of Python, we also introduce the -``typing.Unpack`` type operator for use in existing versions of Python. ``Unpack`` -takes a single type variable tuple argument, and behaves identically to the star -operator, but without requiring a syntax change. In any place you would normally -write ``*Ts``, you can also write ``Unpack[Ts]``. +Note that in the previous example, the ``shape`` argument to ``__init__`` +was annotated as ``Tuple[*Shape]``. Why is this necessary - if ``Shape`` +behaves like ``Tuple[T1, T2, ...]``, couldn't we have annotated the ``shape`` +argument as ``Shape`` directly? + +This is, in fact, deliberately not possible: type variable tuples must +*always* be used unpacked (that is, prefixed by the star operator). This is +for two reasons: + +* To avoid potential confusion about whether to use a type variable tuple + in a packed or unpacked form ("Hmm, should I do ``-> Shape``, + or ``-> Tuple[Shape]``, or ``-> Tuple[*Shape]``...?") +* To improve readability: the star also functions as an explicit visual + indicator that the type variable tuple is not a normal type variable. + +``Unpack`` for Backwards Compatibility +'''''''''''''''''''''''''''''''''''''' + +Note that the use of the star operator in this context requires a grammar change, +and is therefore available only in new versions of Python. To enable use of type +variable tuples in older versions of Python, we introduce the ``Unpack`` type +operator that can be used in place of the star operator to unpack type variable tuples: + +:: + + # Unpacking using the star operator in new versions of Python + class Array(Generic[*Shape]): ... + + # Unpacking using ``Unpack`` in older versions of Python + class Array(Generic[Unpack[Shape]]): ... + +Variance, Type Constraints and Type Bounds: Not (Yet) Supported +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +To keep this PEP minimal, ``TypeVarTuple`` does not yet support specification of: + +* Variance (e.g. ``TypeVar('T', covariant=True)``) +* Type constraints (``TypeVar('T', int, float)``) +* Type bounds (``TypeVar('T', bound=ParentClass)``) + +We leave the decision of how these arguments should behave to a future PEP, when variadic generics have been tested in the field. As of this PEP, type variable tuples are +**invariant**. + +Behaviour when Type Parameters are not Specified +'''''''''''''''''''''''''''''''''''''''''''''''' + +When a generic class parameterised by a type variable tuple is used without +any type parameters, it behaves as if its type parameters are ``Any, ...`` +(an arbitrary number of ``Any``): + +:: + + def takes_array_of_any_rank(arr: Array): ... + + x: Array[Height, Width] + takes_array_of_any_rank(x) # Valid + y: Array[Time, Height, Width] + takes_array_of_any_rank(y) # Also valid + +This enables gradual typing: existing functions with arguments annotated as being, +for example, a plain ``tf.Tensor``, will still be valid even if called with +a parameterised ``Tensor[Height, Width]``. + +Type Variable Tuples Must Have Known Length +''''''''''''''''''''''''''''''''''''''''''' + +Note that in the ``takes_array_of_any_rank`` example in the previous section, +``Array`` behaved as if it were ``Tuple[int, ...]``. This situation - when +type parameters are not specified - is the *only* case when a type variable +tuple may be bound to an unknown-length type. That is: + +:: + + def foo(x: Tuple[*Ts]): ... + + x: Tuple[float, ...] + foo(x) # NOT valid; Ts would be bound to ``Tuple[float, ...]`` + +(If this is confusing - didn't we say that type variable tuples are a stand-in +for an *arbitrary* number of types? - note the difference between the +length of the type variable tuple *itself*, and the length of the type it is +*bound* to. Type variable tuples themselves can be of arbitrary length - +that is, they can be bound to ``Tuple[int]``, ``Tuple[int, int]``, and +so on - but the length of the types they are bound to must be of known length - +that is, ``Tuple[int, int]``, but not ``Tuple[int, ...]``.) + +Type Variable Tuple Equality +'''''''''''''''''''''''''''' + +If the same ``TypeVarTuple`` instance is used in multiple places in a signature +or class, a valid type inference might be to bind the ``TypeVarTuple`` to +a ``Tuple`` of a ``Union`` of types: + +:: + + def foo(arg1: Tuple[*Ts], arg2: Tuple[*Ts]) + + a = (0,) + b = ('0',) + foo(a, b) # Can Ts be bound to Tuple[int | str]? + +We do *not* allow this; type unions may *not* appear within the ``Tuple``. +If a type variable tuple appears in multiple places in a signature, +the types must match exactly: + +:: + + def pointwise_multiply( + x: Array[*Shape], + y: Array[*Shape] + ) -> Array[*Shape]: ... + + x: Array[Height] + y: Array[Width] + z: Array[Height, Width] + pointwise_multiply(x, x) # Valid + pointwise_multiply(x, y) # Error + pointwise_multiply(x, z) # Error + +Multiple Type Variable Tuples: Not Allowed +'''''''''''''''''''''''''''''''''''''''''' + +As of this PEP, only a single type variable tuple may appear in a type parameter list: + +:: + + class Array(Generic[*Ts1, *Ts2]): ... # Error + +(``Union`` is the one exception to this rule; see `Type Variable Tuples with ``Union```.) + +Type Prefixing +-------------- + +Type variable tuples don't have to be alone; normal types can be +prefixed to them: + +:: + + Shape = TypeVarTuple('Shape') + Batch = NewType('Batch', int) + + def add_batch_axis(x: Array[*Shape]) -> Array[Batch, *Shape]: ... + def del_batch_axis(x: Array[Batch, *Shape]) -> Array[*Shape]: ... + + x: Array[Height, Width] + y = add_batch(x) # Inferred type is Array[Batch, Height, Width] + z = del_batch(y) # Array[Height, Width] + +Normal ``TypeVar`` instances can also be prefixed: + +:: + + T = TypeVar('T') + Ts = TypeVarTuple('Ts') + + def prefix_tuple( + x: T, + y: Tuple[*Ts] + ) -> Tuple[T, *Ts]: ... + + z = prefix_tuple(x=0, y=(True, 'a')) + # Inferred type of z is Tuple[int, bool, str] + +As of this PEP - that is, we may expand the flexibility of concatenation in future PEPs - prefixing is the only form of concatenation supported. (That is, the type variable tuple must appear last in the type parameter list.) ``*args`` as a Type Variable Tuple -'''''''''''''''''''''''''''''''''' +---------------------------------- PEP 484 states that when a type annotation is provided for ``*args``, each argument must be of the type annotated. That is, if we specify ``*args`` to be type ``int``, then *all* arguments must be of type ``int``. This limits our ability to specify the type signatures of functions that take heterogeneous argument types. -If ``*args`` is annotated as an unpacked type variable tuple, however, the -types of the individual arguments become the types in the type variable tuple: +If ``*args`` is annotated as a type variable tuple, however, the types of the +individual arguments become the types in the type variable tuple: :: + + Ts = TypeVarTuple('Ts') - def args_to_tuple(*args: *Ts) -> Ts: ... + def args_to_tuple(*args: *Ts) -> Tuple[*Ts]: ... args_to_tuple(1, 'a') # Inferred type is Tuple[int, str] -Note that the type variable tuple must be unpacked in order for this new -behaviour to apply. If the type variable tuple is not unpacked, the old -behaviour still applies: +If no arguments are passed, the type variable tuple behaves like an +empty tuple, ``Tuple[()]``. + +Note that, in keeping with the rule that type variable tuples must always +be used unpacked, annotating ``*args`` as being a plain type variable tuple +instance is *not* allowed: :: - # *All* arguments must be of type Tuple[T1, T2], - # where T1 and T2 are the same types for all arguments - def foo(*args: Ts) -> Ts: ... + def foo(*args: Ts): ... # NOT valid - x: Tuple[int, str] - y: Tuple[int, str] - foo(x, y) # Valid +Also note that if the type variable tuple is wrapped in a ``Tuple``, +the old behaviour still applies: all arguments must be a ``Tuple`` +parameterised with the same types. - z: Tuple[bool] - foo(x, z) # Not valid +:: + + def foo(*args: Tuple[*Ts]): ... + + foo((0,), (1,)) # Valid + foo((0,), (1, 2)) # Error + foo((0,), ('1',)) # Error + +Following `Type Variable Tuples Must Have Finite Length When Bound`, note +that the following should *not* type-check as valid (even though it is, of +course, valid at runtime): + +:: + + def foo(*args: Tuple[*Ts]): ... + + def bar(x: Tuple[int, ...]): + foo(*x) # NOT valid Finally, note that a type variable tuple may *not* be used as the type of -``**kwargs``. (We do not yet know of a use-case for this feature, so prefer +``**kwargs``. (We do not yet know of a use case for this feature, so we prefer to leave the ground fresh for a potential future PEP.) :: # NOT valid - def foo(**kwargs: Ts): ... def foo(**kwargs: *Ts): ... Type Variable Tuples with ``Callable`` -'''''''''''''''''''''''''''''''''''''' +-------------------------------------- Type variable tuples can also be used in the arguments section of a ``Callable``: @@ -286,116 +389,98 @@ Type variable tuples can also be used in the arguments section of a :: class Process: - def __init__(target: Callable[[*Ts], Any], args: Tuple[*Ts]): ... + def __init__( + target: Callable[[*Ts], Any], + args: Tuple[*Ts] + ): ... def func(arg1: int, arg2: str): ... - Process(target=func, args=(0, 'foo')) # Passes type-check - Process(target=func, args=('foo', 0)) # Fails type-check + Process(target=func, args=(0, 'foo')) # Valid + Process(target=func, args=('foo', 0)) # Error + +However, note that as of this PEP, if a type variable tuple does appear in +the arguments section of a ``Callable``, it must appear alone. +That is, `Type Prefixing` is not supported in the context of ``Callable``. +(Use cases where this might otherwise be desirable are likely covered through use +of either a `ParamSpec` from PEP 612, or a type variable tuple in the `__call__` +signature of a callback protocol from PEP 544.) Type Variable Tuples with ``Union`` -''''''''''''''''''''''''''''''''''' +----------------------------------- -Finally, type variable tuples can be used with ``Union``: +Type variable tuples can also be used with ``Union``: :: - def f(*args: *Ts) -> Union[*Ts]: + def f(*args: Tuple[*Ts]) -> Union[*Ts]: return random.choice(args) f(1, 'foo') # Inferred type is Union[int, str] -If the type variable tuple is empty (e.g. if we had ``*args: *Ts`` +More than one type variable tuple may appear in the the parameter list +to ``Union``: + +:: + + def cond_random_choice( + cond: bool, + cond_true: Tuple[*Ts1], + cond_false: Tuple[*Ts2] + ) -> Union[*Ts1, *Ts2]: + if cond: + return random.choice(cond_true) + else: + return random.choice(cond_false) + + # Inferred type is Union[int, str, float] + cond_random_choice(True, (1, 'foo'), (0.0, 'bar')) + +If the type variable tuple is empty (e.g. if we had ``*args: Tuple[*Ts]`` and didn't pass any arguments), the type checker should raise an error on the ``Union`` (matching the behaviour of ``Union`` at runtime, which requires at least one type argument). -``Map`` +Aliases ------- -To enable typing of functions such as ``map`` and ``zip``, we introduce the -``Map`` type operator. Not to be confused with the existing operator -``typing.Mapping``, ``Map`` is analogous to ``map``, but for types: +Generic aliases can be created using a type variable tuple in +a similar way to regular type variables: :: - from typing import Map + IntTuple = Tuple[int, *Ts] + IntTuple[float, bool] # Equivalent to Tuple[int, float, bool] - def args_to_lists(*args: *Ts) -> Map[List, Ts]: ... - - args_to_lists(1, 'a') # Inferred type is Tuple[List[int], List[str]] - -``Map`` takes two operands. The first operand is a parameterizable -type (or type alias [#type_aliases]_) such as ``Tuple``, ``List``, or a -user-defined generic class. The second operand is a type variable tuple. -The result of ``Map`` is a ``Tuple``, where the Nth type in the ``Tuple`` is -the first operand parameterized by the Nth type in the type variable tuple. - -Because ``Map`` returns a parameterized ``Tuple``, it can be used anywhere -that a type variable tuple would be. For example, as the type of ``*args``: - -:: - - # Equivalent to 'arg1: List[T1], arg2: List[T2], ...' - def foo(*args: *Map[List, Ts]): ... - # Ts is bound to Tuple[int, str] - foo([1], ['a']) - -As a return type: +As this example shows, all type parameters passed to the alias are +bound to the type variable tuple. If no type parameters are given, +or if an explicitly empty list of type parameters are given, +type variable tuple in the alias is simply ignored: :: - # Equivalent to '-> Tuple[List[T1], List[T2], ...]' - def bar(*args: *Ts) -> Map[List, Ts]: ... - # Ts is bound to Tuple[float, bool] - # Inferred type is Tuple[List[float], List[bool]] - bar(1.0, True) + # Both equivalent to Tuple[int] + IntTuple + IntTuple[()] -And as an argument type: +Normal ``TypeVar`` instances can also be used in such aliases: :: - # Equivalent to 'arg: Tuple[List[T1], List[T2], ...]' - def baz(arg: Map[List, Ts]): ... - # Ts is bound to Tuple[bool, bool] - baz(([True], [False])) + T = TypeVar('T') + Foo = Tuple[T, *Ts] -``map`` and ``zip`` -''''''''''''''''''' + # T is bound to `int`; Ts is bound to `bool, str` + Foo[int, bool, str] -``Map`` allows us to specify the signature of ``map`` as: - -:: - - Ts = TypeVarTuple('Ts') - R = TypeVar(R) - - def map(func: Callable[[*Ts], R], - *iterables: *Map[Iterable, Ts]) -> Iterable[R]: ... - - def func(int, str) -> float: ... - # Ts is bound to Tuple[int, str] - # Map[Iterable, Ts] is Iterable[int], Iterable[str] - # Therefore, iter1 must be type Iterable[int], - # and iter2 must be type Iterable[str] - map(func, iter1, iter2) - -Similarly, we can specify the signature of ``zip`` as: - -:: - - def zip(*iterables: *Map[Iterable, Ts]) -> Iterator[Ts]): ... - - l1: List[int] - l2: List[str] - zip(l1, l2) # Iterator[Tuple[int, str]] +Note that the same rules for `Type Prefixing` apply for aliases. +In particular, only one ``TypeVarTuple`` may occur within an alias, +and the ``TypeVarTuple`` must be at the end of the alias. Overloads for Accessing Individual Types ---------------------------------------- -``Map`` allows us to operate on types in a bulk fashion. For situations where we -require access to each individual type, overloads can be used with individual -``TypeVar`` instances in place of the type variable tuple: +For situations where we require access to each individual type, overloads can be used with individual ``TypeVar`` instances in place of the type variable tuple: :: @@ -404,7 +489,7 @@ require access to each individual type, overloads can be used with individual Axis2 = TypeVar('Axis2') Axis3 = TypeVar('Axis3') - class Array(Generic[*Shape]): ... + class Array(Generic[Shape]): ... @overload def transpose( @@ -421,232 +506,6 @@ overloads for each possible rank is, of course, a rather cumbersome solution. However, it's the best we can do without additional type manipulation mechanisms, which are beyond the scope of this PEP.) -Concatenating Other Types to a Type Variable Tuple --------------------------------------------------- - -If an unpacked type variable tuple appears with other types in the same type parameter -list, the effect is to concatenate those types with the types in the type variable -tuple. For example, concatenation in a function return type: - -:: - - Batch = NewType('int') - Height = NewType('int') - Width = NewType('int') - - class Array(Generic[*Shape]): ... - - def add_batch(x: Array[*Shape]) -> Array[Batch, *Shape]: ... - - x: Array[Height, Width] - y = add_batch(x) # Inferred type is Array[Batch, Height, Width] - -In function argument types: - -:: - - def batch_sum(x: Array[Batch, *Shape]) -> Array[*Shape]: ... - - x: Array[Batch, Height, Width] - y = batch_sum(x) # Inferred type is Array[Height, Width] - -And in class type parameters: - -:: - - class BatchArray(Generic[Batch, *Shape]): - def sum(self) -> Array[*Shape]: ... - - x: BatchArray[Batch, Height, Width] - y = x.sum() # Inferred type is Array[Height, Width] - -Concatenation can involve both prefixing and suffixing, and -can include an arbitrary number of types: - -:: - - def foo(x: Tuple[*Ts]) -> Tuple[int, str, *Ts, bool]: ... - -It is also possible to concatenate type variable tuples with regular -type variables: - -:: - - T = TypeVar('T') - - def first_axis_sum(x: Array[T, *Shape]) -> Array[*Shape]: ... - - x: Array[Time, Height, Width] - y = first_axis_sum(x) # Inferred type is Array[Height, Width] - -Finally, concatenation can also occur in the argument list to ``Callable``: - -:: - - def f(func: Callable[[int, *Ts], Any]) -> Tuple[*Ts]: ... - - def foo(int, str, float): ... - def bar(str, int, float): ... - - f(foo) # Valid; inferred type is Tuple[str, float] - f(bar) # Not valid - -And in ``Union``: - -:: - - def f(*args: *Ts) -> Union[*Ts, float]: ... - - f(0, 'spam') # Inferred type is Union[int, str, float] - -Concatenating Multiple Type Variable Tuples -------------------------------------------- - -We can also concatenate *multiple* type variable tuples, but only in cases -where the types bound to each type variable tuple can be inferred -unambiguously. Note that this is not always the case: - -:: - - # Type checker should raise an error on definition of func; - # how would we know which types are bound to Ts1, and which - # are bound to Ts2? - def func(ham: Tuple[*Ts1, *Ts2]): ... - - # Ts1 = Tuple[int, str], Ts2 = Tuple[bool]? - # Or Ts1 = Tuple[int], Ts2 = Tuple[str, bool]? - ham: Tuple[int, str, bool] - func(ham) - -In general, some kind of extra constraint is necessary in order -for the ambiguity to be resolved. This is usually provided by -an un-concatenated usage of the type variable tuple elsewhere in -the same signature. - -For example, resolving ambiguity in an argument: - -:: - - def func(ham: Tuple[*Ts1, *Ts2], spam: Ts2): ... - - # Ts1 is bound to Tuple[int], Ts2 to Tuple[str, bool] - ham: Tuple[int, str, bool] - spam: Tuple[str, bool] - func(ham, spam) - -In a return type: - -:: - - def func(ham: Ts1, spam: Ts2) -> Tuple[*Ts1, *Ts2]): ... - - ham: Tuple[int] - spam: Tuple[str, bool] - # Return type is Tuple[int, str, bool] - func(ham, spam) - -Note, however, that the same cannot be done with generic classes: - -:: - - # No way to add extra constraints about Ts1 and Ts2, - # so this is not valid - class C(Generic[*Ts1, *Ts2]): ... - -Generics in Multiple Type Variable Tuples ------------------------------------------ - -If we *do* wish to use multiple type variable tuples in a type signature -that would otherwise not resolve the ambiguity, it is also possible -to make the type bindings explicit by using a type variable tuple directly, -without unpacking it. When then instantiating, for example, the class in -question, the types corresponding to each type variable tuple must -be wrapped in a ``Tuple``: - -:: - - class C(Generic[Ts1, Ts2]): ... - - # Ts1 = Tuple[int, str] - # Ts2 = Tuple[bool] - c: C[Tuple[int, str], Tuple[bool]] = C() - -Similarly for functions: - -:: - - def foo(x: Tuple[Ts1, Ts2]): ... - - # Ts1 = Tuple[int, float] - # Ts2 = Tuple[bool] - x: Tuple[Tuple[int, float], Tuple[bool]] - foo(x) - -Aliases -------- - -Generic aliases can be created using a type variable tuple in -a similar way to regular type variables: - -:: - - IntTuple = Tuple[int, *Ts] - IntTuple[float, bool] # Equivalent to Tuple[int, float, bool] - -As this example shows, all type arguments passed to the alias are -bound to the type variable tuple. If no type arguments are given, -the type variable tuple holds no types: - -:: - - IntTuple # Equivalent to Tuple[int] - -Type variable tuples can also be used without unpacking: - -:: - - IntTuple = Tuple[int, Ts] - IntTuple[float, bool] # Equivalent to Tuple[int, Tuple[float, bool]] - IntTuple # Tuple[int, Tuple[]] - -At most a single distinct type variable tuple can occur in an alias: - -:: - - # Invalid - Foo = Tuple[Ts1, int, Ts2] - # Why? Because there would be no way to decide which types should - # be bound to which type variable tuple: - Foo[float, bool, str] - # Equivalent to Tuple[float, bool, int, str]? - # Or Tuple[float, int, bool, str]? - -The same type variable tuple may be used multiple times, however: - -:: - - Bar = Tuple[*Ts, *Ts] - Bar[int, float] # Equivalent to Tuple[int, float, int, float] - -Finally, type variable tuples can be used in combination with -normal type variables. In this case, the number of type arguments must -be equal to or greater than the number of distinct normal type variables: - -:: - - Baz = Tuple[T1, *Ts, T2, T1] - - # T1 bound to int, T2 bound to bool, Ts empty - # Equivalent to Tuple[int, bool, int] - Baz[int, bool] - - # T1 bound to int - # Ts bound to Tuple[float, bool] - # T2 bound to str - # So equivalent to Tuple[int, float, bool, str, int] - Baz[int, float, bool, str] - - An Ideal Array Type: One Possible Example ========================================= @@ -693,6 +552,11 @@ in a complete array type. # E.g. Array1D[np.uint8] Array1D = Array[DataType, Ndim[Literal[1]]] +Final Notes +=========== + +**Slice expressions**: type variable tuples may *not* appear in slice expressions. + Rationale and Rejected Ideas ============================ @@ -710,40 +574,8 @@ by simply defining aliases for each possible number of type parameters: However, this seems somewhat clumsy - it requires users to unnecessarily pepper their code with 1s, 2s, and so on for each rank necessary. -Naming of ``Map`` ------------------ - -One downside to the name ``Map`` is that it might suggest a hash map. We -considered a number of different options for the name of this operator. - -* ``ForEach``. This is rather long, and we thought might imply a side-effect. -* ``Transform``. The meaning of this isn't obvious enough at first glance. -* ``Apply``. This is inconsistent with ``apply``, an older Python function - which enabled conversion of iterables to arguments before the star - operator was introduced. - -In the end, we decided that ``Map`` was good enough. - -Nesting ``Map`` ---------------- - -Since the result of ``Map`` is a parameterised ``Tuple``, it should be -possible to use the output of a ``Map`` as the input to another ``Map``: - -:: - - Map[Tuple, Map[List, Ts]] - -If ``Ts`` here were bound to ``Tuple[int, str]``, the result of the -inner ``Map`` would be ``Tuple[List[int], List[str]]``, so the result -of the outer map would be ``Tuple[Tuple[List[int]], Tuple[List[str]]]``. - -We chose not to highlight this fact because of a) how confusing it is, -and b) lack of a specific use-case. Whether to support nested ``Map`` -is left to the implementation. - -Naming of ``TypeVarTuple`` --------------------------- +Construction of ``TypeVarTuple`` +-------------------------------- ``TypeVarTuple`` began as ``ListVariadic``, based on its naming in an early implementation in Pyre. @@ -752,12 +584,15 @@ We then changed this to ``TypeVar(list=True)``, on the basis that a) it better emphasises the similarity to ``TypeVar``, and b) the meaning of 'list' is more easily understood than the jargon of 'variadic'. -We finally settled on ``TypeVarTuple`` based on the justification -that c) this emphasises the tuple-like behaviour, and d) type variable -tuples are a sufficiently different kind of thing to regular -type variables that we may later wish to support keyword arguments -to its constructor that should not be supported by regular -type variables (such as ``arbitrary_len`` [#arbitrary_len]_). +Once we'd decided that a variadic type variable should behave like a ``Tuple``, +we also considered ``TypeVar(bound=Tuple)``, which is similarly intuitive +and accomplishes most what we wanted without requiring any new arguments to +``TypeVar``. However, we realised this may constrain us in the future, if +for example we want type bounds or variance to function slightly differently +for variadic type variables than what the semantics of ``TypeVar`` might +otherwise imply. Also, we may later wish to support arguments that should not be supported by regular type variables (such as ``arbitrary_len`` [#arbitrary_len]_). + +We therefore settled on ``TypeVarTuple``. Backwards Compatibility ======================= @@ -776,9 +611,6 @@ TODO Footnotes ========== -.. [#hkt] A third potential use is in enabling higher-kinded types that take - an arbitrary number of type operands, but we do not discuss this use - here. .. [#batch] 'Batch' is machine learning parlance for 'a number of'. @@ -791,11 +623,6 @@ Footnotes shape begins with 'time × batch', then ``videos_batch[1][0]`` would select the same frame. -.. [#type_aliases] For example, in ``asyncio`` [#asyncio]_, it is convenient - to define a type alias - ``_FutureT = Union[Future[_T], Generator[Any, None, _T], Awaitable[_T]]``. - We should be able to apply ``Map`` to such aliases - e.g. ``Map[_FutureT, Ts]``. - References ========== @@ -810,10 +637,6 @@ References .. [#syntax-proposal] Shape annotation syntax proposal: https://docs.google.com/document/d/1But-hjet8-djv519HEKvBN6Ik2lW3yu0ojZo6pG9osY/edit -.. [#zip-sig] ``typeshed/builtins.pyi``: https://github.com/python/typeshed/blob/27dfbf68aaffab4f1ded7dc1b96f6f82f536a09d/stdlib/2and3/builtins.pyi#L1710-L1733 - -.. [#asyncio] ``typeshed/asyncio/tasks.pyi``: https://github.com/python/typeshed/blob/193c7cb93283ad4ca2a65df74c565e56bfe72b7e/stdlib/3/asyncio/tasks.pyi#L45-L154 - .. [#arbitrary_len] Discussion on Python typing-sig mailing list: https://mail.python.org/archives/list/typing-sig@python.org/thread/SQVTQYWIOI4TIO7NNBTFFWFMSMS2TA4J/ @@ -825,8 +648,7 @@ Thank you to **Alfonso Castaño**, **Antoine Pitrou**, **Bas v.B.**, **David Fos **Sergei Lebedev** and **Vladimir Mikulik** for helpful feedback and suggestions on drafts of this PEP. -Thank you especially to **Lucio**, for suggesting the star syntax, which has made -multiple aspects of this proposal much more concise and intuitive. +Thank you especially to **Lucio**, for suggesting the star syntax, which has made multiple aspects of this proposal much more concise and intuitive. Resources ========= @@ -865,4 +687,3 @@ CC0-1.0-Universal license, whichever is more permissive. fill-column: 70 coding: utf-8 End: -