PEP 646: Update based on typing-sig discussions (#1781)
* PEP 646: Simplify based on typing-sig discussions Feature removals: * Remove `Map` (to be added in a separate PEP) * Remove support for all forms of concatenation other than simply prefixing of a single `TypeVarTuple` (fancier forms of concatenation to also be added in a separate PEP) Semantic changes: * Try switching `Ts` back to mean the 'unpacked' version (such that `Ts = int, str; Tuple[Ts] = Tuple[int, str]`) to see whether everything still works. (Might revert this, pending future discussion in typing-sig.) * PEP 646: Clarify that prefixing is not allowed in Callable * Update PEP 646 based on discussions in typing-sig Semantic changes: * Switch back to `Ts` meaning types packed in a `Tuple` * Disallow unpacked usages of `Ts` * Unparameterised `Tensor` behaves like `Tensor[Any, ...]` Clarifications: * `TypeVarTuple`s are invariant * `TypeVarTuple`s can not be bound to unknown-length types (e.g. `Tuple[int, ...]` * `TypeVarTuple`s can not be bound to unions of types * Empty `*args` behaves like `Tuple[()]` * `TypeVarTuple`s can not appear in slices Readability changes: * Reorder some sections for better flow * Update PEP 646 with Eric's suggestions and other tweaks Semantic changes: * Revert to `*args: *Ts` for new behaviour, and `*args: Tuple[*Ts]` for old behaviour Fixes: * Unpacking in Aliases/Ideal Array Type sections * Various typo/grammar fixes Other: * State explicitly that TypeVarTuples can be used in callback protocols
This commit is contained in:
Matthew Rahtz
GitHub
parent
6f091c5969
commit
68794b15c3
821
pep-0646.rst
821
pep-0646.rst
|
|
@ -1,4 +1,4 @@
|
|||
PEP: 0646
|
||||
PEP: 646
|
||||
Title: Variadic Generics
|
||||
Author: Mark Mendoza <mendoza.mark.a@gmail.com>,
|
||||
Matthew Rahtz <mrahtz@google.com>,
|
||||
|
|
@ -26,20 +26,6 @@ to catch shape-related bugs in code that uses these libraries.
|
|||
Motivation
|
||||
==========
|
||||
|
||||
There are two main use-cases for variadic generics. [#hkt]_
|
||||
|
||||
The primary motivation is to enable typing of array shapes in numerical
|
||||
computing libraries such as NumPy and TensorFlow. This is the use-case
|
||||
much of the PEP will focus on.
|
||||
|
||||
Additionally, variadic generics allow us to concisely specify the type
|
||||
signature of ``map`` and ``zip``.
|
||||
|
||||
We discuss each of these motivations below.
|
||||
|
||||
Array Shapes
|
||||
-------------
|
||||
|
||||
In the context of numerical computation with libraries such as NumPy and
|
||||
TensorFlow, the *shape* of arguments is often just as important as the
|
||||
argument *type*. For example, consider the following function which converts a
|
||||
|
|
@ -87,56 +73,10 @@ and so on throughout their code) and for the authors of array libraries (who wou
|
|||
Variadic generics are necessary for a ``Array`` that is generic in an arbitrary
|
||||
number of axes to be cleanly defined as a single class.
|
||||
|
||||
``map`` and ``zip``
|
||||
-------------------
|
||||
|
||||
PEP 612 [#pep-612]_ introduced ``ParamSpec``, enabling parameter types of one
|
||||
callable to be forwarded to another callable. However, in many cases we actually
|
||||
wish to *transform* parameter types before using them elsewhere in the
|
||||
signature.
|
||||
|
||||
Consider, for example, the signature of ``map`` for a particular choice of
|
||||
function and iterables:
|
||||
|
||||
::
|
||||
|
||||
def func(int, str) -> float: ...
|
||||
iter1: List[int]
|
||||
iter2: List[str]
|
||||
|
||||
def map(func: Callable[[int, str], float],
|
||||
iter1: Iterable[int],
|
||||
iter2: Iterable[str]) -> Iterable[float]: ...
|
||||
|
||||
Note that the number of iterables passed to ``map`` is dependent
|
||||
on the supplied function, and that the types of those iterables
|
||||
must correspond to the types of supplied function's arguments.
|
||||
|
||||
A similar example is ``zip``:
|
||||
|
||||
::
|
||||
|
||||
iter1: List[int]
|
||||
iter2: List[str]
|
||||
|
||||
def zip(iter1: Iterable[int],
|
||||
iter2: Iterable[str]) -> Iterable[Tuple[int, str]]: ...
|
||||
|
||||
Neither of these signatures can be specified in the general case using
|
||||
existing typing mechanisms. The signature of ``zip``, for example, is
|
||||
currently specified [#zip-sig]_ with a number of overloads.
|
||||
|
||||
Specification
|
||||
=============
|
||||
|
||||
In order to support the above use-cases, we introduce:
|
||||
|
||||
* ``TypeVarTuple``, serving as a placeholder not for a single type but
|
||||
for an *arbitrary* number of types, and behaving like a number of
|
||||
``TypeVar`` instances packed in a ``Tuple``.
|
||||
* A new use for the star operator: unpacking of each individual type
|
||||
from a ``TypeVarTuple``.
|
||||
* Two new type operators, ``Unpack`` and ``Map``.
|
||||
In order to support the above use cases, we introduce ``TypeVarTuple``. This serves as a placeholder not for a single type but for an *arbitrary* number of types, and behaving like a number of ``TypeVar`` instances packed in a ``Tuple``.
|
||||
|
||||
These are described in detail below.
|
||||
|
||||
|
|
@ -155,130 +95,293 @@ Type variable tuples are created with:
|
|||
|
||||
Ts = TypeVarTuple('Ts')
|
||||
|
||||
A type variable tuple behaves in a similar way to a parameterized ``Tuple``.
|
||||
For example, in a generic object instantiated with type parameters
|
||||
``int`` and ``str``, ``Ts`` is equivalent to ``Tuple[int, str]``.
|
||||
Type variable tuples behave like a number of individual type variables packed in a
|
||||
``Tuple``. To understand this, consider the following example:
|
||||
|
||||
::
|
||||
|
||||
Shape = TypeVarTuple('Shape')
|
||||
|
||||
class Array(Generic[*Shape]): ...
|
||||
|
||||
Height = NewType('Height', int)
|
||||
Width = NewType('Width', int)
|
||||
x: Array[Height, Width] = Array()
|
||||
|
||||
The ``Shape`` type variable tuple here behaves like ``Tuple[T1, T2]``,
|
||||
where ``T1`` and ``T2`` are type variables. To use these type variables
|
||||
as type parameters of ``Array``, we must **unpack** the type variable tuple using
|
||||
the star operator: ``*Shape``. The signature of ``Array`` then behaves
|
||||
as if we had simply written ``class Array(Generic[T1, T2]): ...``.
|
||||
|
||||
In contrast to ``Generic[T1, T2]``, however, ``Generic[*Shape]`` allows
|
||||
us to parameterise the class with an *arbitrary* number of type parameters.
|
||||
That is, in addition to being able to define rank-2 arrays such as
|
||||
``Array[Height, Width]``, we could also define rank-3 arrays, rank-4 arrays,
|
||||
and so on:
|
||||
|
||||
::
|
||||
|
||||
Time = NewType('Time', int)
|
||||
Batch = NewType('Batch', int)
|
||||
y: Array[Batch, Height, Width] = Array()
|
||||
z: Array[Time, Batch, Height, Width] = Array()
|
||||
|
||||
Type variable tuples can be used anywhere a normal ``TypeVar`` can.
|
||||
For example, in class definitions, function signatures, and variable annotations:
|
||||
This includes class definitions, as shown above, as well as function
|
||||
signatures and variable annotations:
|
||||
|
||||
::
|
||||
|
||||
Shape = TypeTupleVar('Shape')
|
||||
|
||||
class Array(Generic[Shape]):
|
||||
|
||||
def __init__(self, shape: Shape):
|
||||
self.shape: Shape = shape
|
||||
|
||||
def __abs__(self) -> Array[Shape]: ...
|
||||
|
||||
def __add__(self, other: Array[Shape]) -> Array[Shape]: ...
|
||||
|
||||
Height = NewType('Height', int)
|
||||
Width = NewType('Width', int)
|
||||
shape = (Height(480), Width(640))
|
||||
x: Array[Tuple[Height, Width]] = Array(shape)
|
||||
x.shape # Inferred type is Tuple[Height, Width]
|
||||
y = abs(x) # Array[Tuple[Height, Width]]
|
||||
z = x + y # Array[Tuple[Height, Width]]
|
||||
|
||||
Variance and ``bound``: Not (Yet) Supported
|
||||
'''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
To keep this PEP minimal, ``TypeTupleVar`` does not yet support
|
||||
the ``bound`` argument or specification of variance, as ``TypeVar``
|
||||
does. We leave the decision of how these arguments should be implemented
|
||||
to a future PEP, when use-cases for variadic generics have been
|
||||
explored more in practice.
|
||||
|
||||
Unpacking: Star Operator
|
||||
''''''''''''''''''''''''
|
||||
|
||||
Note that the fully-parameterised type of ``Array`` above is
|
||||
rather verbose. Wouldn't it be easier if we could just write
|
||||
``Array[Height, Width]``?
|
||||
|
||||
To enable this, we introduce a new function for the star operator:
|
||||
to 'unpack' type variable tuples. When unpacked, a type variable tuple
|
||||
behaves as if its component types had been written
|
||||
directly into the signature, rather than being wrapped in a ``Tuple``.
|
||||
|
||||
Rewriting the ``Array`` class using an unpacked type variable
|
||||
tuple, we can instead write:
|
||||
|
||||
::
|
||||
|
||||
Shape = TypeTupleVar('Shape')
|
||||
|
||||
class Array(Generic[*Shape]):
|
||||
|
||||
def __init__(self, shape: Shape):
|
||||
self.shape: Shape = shape
|
||||
def __init__(self, shape: Tuple[*Shape]):
|
||||
self._shape: Tuple[*Shape] = shape
|
||||
|
||||
def __add__(self, other: Array[*Shape]) -> Array[*Shape]: ...
|
||||
def get_shape(self) -> Tuple[*Shape]:
|
||||
return self._shape
|
||||
|
||||
def __abs__(self) -> Array[*Shape]: ...
|
||||
|
||||
def __add__(self, other: Array[*Shape]) -> Array[*Shape]) ...
|
||||
|
||||
shape = (Height(480), Width(640))
|
||||
x: Array[Height, Width] = Array(shape)
|
||||
x.shape # Inferred type is Tuple[Height, Width]
|
||||
z = x + x # Array[Height, Width]
|
||||
y = abs(x) # Inferred type is Array[Height, Width]
|
||||
z = x + x # ... is Array[Height, Width]
|
||||
|
||||
Unpacking: ``Unpack`` Operator
|
||||
''''''''''''''''''''''''''''''
|
||||
Type Variable Tuples Must Always be Unpacked
|
||||
''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Because the new use of the star operator requires a syntax change and is
|
||||
therefore incompatible with previous versions of Python, we also introduce the
|
||||
``typing.Unpack`` type operator for use in existing versions of Python. ``Unpack``
|
||||
takes a single type variable tuple argument, and behaves identically to the star
|
||||
operator, but without requiring a syntax change. In any place you would normally
|
||||
write ``*Ts``, you can also write ``Unpack[Ts]``.
|
||||
Note that in the previous example, the ``shape`` argument to ``__init__``
|
||||
was annotated as ``Tuple[*Shape]``. Why is this necessary - if ``Shape``
|
||||
behaves like ``Tuple[T1, T2, ...]``, couldn't we have annotated the ``shape``
|
||||
argument as ``Shape`` directly?
|
||||
|
||||
This is, in fact, deliberately not possible: type variable tuples must
|
||||
*always* be used unpacked (that is, prefixed by the star operator). This is
|
||||
for two reasons:
|
||||
|
||||
* To avoid potential confusion about whether to use a type variable tuple
|
||||
in a packed or unpacked form ("Hmm, should I do ``-> Shape``,
|
||||
or ``-> Tuple[Shape]``, or ``-> Tuple[*Shape]``...?")
|
||||
* To improve readability: the star also functions as an explicit visual
|
||||
indicator that the type variable tuple is not a normal type variable.
|
||||
|
||||
``Unpack`` for Backwards Compatibility
|
||||
''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Note that the use of the star operator in this context requires a grammar change,
|
||||
and is therefore available only in new versions of Python. To enable use of type
|
||||
variable tuples in older versions of Python, we introduce the ``Unpack`` type
|
||||
operator that can be used in place of the star operator to unpack type variable tuples:
|
||||
|
||||
::
|
||||
|
||||
# Unpacking using the star operator in new versions of Python
|
||||
class Array(Generic[*Shape]): ...
|
||||
|
||||
# Unpacking using ``Unpack`` in older versions of Python
|
||||
class Array(Generic[Unpack[Shape]]): ...
|
||||
|
||||
Variance, Type Constraints and Type Bounds: Not (Yet) Supported
|
||||
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
To keep this PEP minimal, ``TypeVarTuple`` does not yet support specification of:
|
||||
|
||||
* Variance (e.g. ``TypeVar('T', covariant=True)``)
|
||||
* Type constraints (``TypeVar('T', int, float)``)
|
||||
* Type bounds (``TypeVar('T', bound=ParentClass)``)
|
||||
|
||||
We leave the decision of how these arguments should behave to a future PEP, when variadic generics have been tested in the field. As of this PEP, type variable tuples are
|
||||
**invariant**.
|
||||
|
||||
Behaviour when Type Parameters are not Specified
|
||||
''''''''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
When a generic class parameterised by a type variable tuple is used without
|
||||
any type parameters, it behaves as if its type parameters are ``Any, ...``
|
||||
(an arbitrary number of ``Any``):
|
||||
|
||||
::
|
||||
|
||||
def takes_array_of_any_rank(arr: Array): ...
|
||||
|
||||
x: Array[Height, Width]
|
||||
takes_array_of_any_rank(x) # Valid
|
||||
y: Array[Time, Height, Width]
|
||||
takes_array_of_any_rank(y) # Also valid
|
||||
|
||||
This enables gradual typing: existing functions with arguments annotated as being,
|
||||
for example, a plain ``tf.Tensor``, will still be valid even if called with
|
||||
a parameterised ``Tensor[Height, Width]``.
|
||||
|
||||
Type Variable Tuples Must Have Known Length
|
||||
'''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
Note that in the ``takes_array_of_any_rank`` example in the previous section,
|
||||
``Array`` behaved as if it were ``Tuple[int, ...]``. This situation - when
|
||||
type parameters are not specified - is the *only* case when a type variable
|
||||
tuple may be bound to an unknown-length type. That is:
|
||||
|
||||
::
|
||||
|
||||
def foo(x: Tuple[*Ts]): ...
|
||||
|
||||
x: Tuple[float, ...]
|
||||
foo(x) # NOT valid; Ts would be bound to ``Tuple[float, ...]``
|
||||
|
||||
(If this is confusing - didn't we say that type variable tuples are a stand-in
|
||||
for an *arbitrary* number of types? - note the difference between the
|
||||
length of the type variable tuple *itself*, and the length of the type it is
|
||||
*bound* to. Type variable tuples themselves can be of arbitrary length -
|
||||
that is, they can be bound to ``Tuple[int]``, ``Tuple[int, int]``, and
|
||||
so on - but the length of the types they are bound to must be of known length -
|
||||
that is, ``Tuple[int, int]``, but not ``Tuple[int, ...]``.)
|
||||
|
||||
Type Variable Tuple Equality
|
||||
''''''''''''''''''''''''''''
|
||||
|
||||
If the same ``TypeVarTuple`` instance is used in multiple places in a signature
|
||||
or class, a valid type inference might be to bind the ``TypeVarTuple`` to
|
||||
a ``Tuple`` of a ``Union`` of types:
|
||||
|
||||
::
|
||||
|
||||
def foo(arg1: Tuple[*Ts], arg2: Tuple[*Ts])
|
||||
|
||||
a = (0,)
|
||||
b = ('0',)
|
||||
foo(a, b) # Can Ts be bound to Tuple[int | str]?
|
||||
|
||||
We do *not* allow this; type unions may *not* appear within the ``Tuple``.
|
||||
If a type variable tuple appears in multiple places in a signature,
|
||||
the types must match exactly:
|
||||
|
||||
::
|
||||
|
||||
def pointwise_multiply(
|
||||
x: Array[*Shape],
|
||||
y: Array[*Shape]
|
||||
) -> Array[*Shape]: ...
|
||||
|
||||
x: Array[Height]
|
||||
y: Array[Width]
|
||||
z: Array[Height, Width]
|
||||
pointwise_multiply(x, x) # Valid
|
||||
pointwise_multiply(x, y) # Error
|
||||
pointwise_multiply(x, z) # Error
|
||||
|
||||
Multiple Type Variable Tuples: Not Allowed
|
||||
''''''''''''''''''''''''''''''''''''''''''
|
||||
|
||||
As of this PEP, only a single type variable tuple may appear in a type parameter list:
|
||||
|
||||
::
|
||||
|
||||
class Array(Generic[*Ts1, *Ts2]): ... # Error
|
||||
|
||||
(``Union`` is the one exception to this rule; see `Type Variable Tuples with ``Union```.)
|
||||
|
||||
Type Prefixing
|
||||
--------------
|
||||
|
||||
Type variable tuples don't have to be alone; normal types can be
|
||||
prefixed to them:
|
||||
|
||||
::
|
||||
|
||||
Shape = TypeVarTuple('Shape')
|
||||
Batch = NewType('Batch', int)
|
||||
|
||||
def add_batch_axis(x: Array[*Shape]) -> Array[Batch, *Shape]: ...
|
||||
def del_batch_axis(x: Array[Batch, *Shape]) -> Array[*Shape]: ...
|
||||
|
||||
x: Array[Height, Width]
|
||||
y = add_batch(x) # Inferred type is Array[Batch, Height, Width]
|
||||
z = del_batch(y) # Array[Height, Width]
|
||||
|
||||
Normal ``TypeVar`` instances can also be prefixed:
|
||||
|
||||
::
|
||||
|
||||
T = TypeVar('T')
|
||||
Ts = TypeVarTuple('Ts')
|
||||
|
||||
def prefix_tuple(
|
||||
x: T,
|
||||
y: Tuple[*Ts]
|
||||
) -> Tuple[T, *Ts]: ...
|
||||
|
||||
z = prefix_tuple(x=0, y=(True, 'a'))
|
||||
# Inferred type of z is Tuple[int, bool, str]
|
||||
|
||||
As of this PEP - that is, we may expand the flexibility of concatenation in future PEPs - prefixing is the only form of concatenation supported. (That is, the type variable tuple must appear last in the type parameter list.)
|
||||
|
||||
``*args`` as a Type Variable Tuple
|
||||
''''''''''''''''''''''''''''''''''
|
||||
----------------------------------
|
||||
|
||||
PEP 484 states that when a type annotation is provided for ``*args``, each argument
|
||||
must be of the type annotated. That is, if we specify ``*args`` to be type ``int``,
|
||||
then *all* arguments must be of type ``int``. This limits our ability to specify
|
||||
the type signatures of functions that take heterogeneous argument types.
|
||||
|
||||
If ``*args`` is annotated as an unpacked type variable tuple, however, the
|
||||
types of the individual arguments become the types in the type variable tuple:
|
||||
If ``*args`` is annotated as a type variable tuple, however, the types of the
|
||||
individual arguments become the types in the type variable tuple:
|
||||
|
||||
::
|
||||
|
||||
Ts = TypeVarTuple('Ts')
|
||||
|
||||
def args_to_tuple(*args: *Ts) -> Ts: ...
|
||||
def args_to_tuple(*args: *Ts) -> Tuple[*Ts]: ...
|
||||
|
||||
args_to_tuple(1, 'a') # Inferred type is Tuple[int, str]
|
||||
|
||||
Note that the type variable tuple must be unpacked in order for this new
|
||||
behaviour to apply. If the type variable tuple is not unpacked, the old
|
||||
behaviour still applies:
|
||||
If no arguments are passed, the type variable tuple behaves like an
|
||||
empty tuple, ``Tuple[()]``.
|
||||
|
||||
Note that, in keeping with the rule that type variable tuples must always
|
||||
be used unpacked, annotating ``*args`` as being a plain type variable tuple
|
||||
instance is *not* allowed:
|
||||
|
||||
::
|
||||
|
||||
# *All* arguments must be of type Tuple[T1, T2],
|
||||
# where T1 and T2 are the same types for all arguments
|
||||
def foo(*args: Ts) -> Ts: ...
|
||||
def foo(*args: Ts): ... # NOT valid
|
||||
|
||||
x: Tuple[int, str]
|
||||
y: Tuple[int, str]
|
||||
foo(x, y) # Valid
|
||||
Also note that if the type variable tuple is wrapped in a ``Tuple``,
|
||||
the old behaviour still applies: all arguments must be a ``Tuple``
|
||||
parameterised with the same types.
|
||||
|
||||
z: Tuple[bool]
|
||||
foo(x, z) # Not valid
|
||||
::
|
||||
|
||||
def foo(*args: Tuple[*Ts]): ...
|
||||
|
||||
foo((0,), (1,)) # Valid
|
||||
foo((0,), (1, 2)) # Error
|
||||
foo((0,), ('1',)) # Error
|
||||
|
||||
Following `Type Variable Tuples Must Have Finite Length When Bound`, note
|
||||
that the following should *not* type-check as valid (even though it is, of
|
||||
course, valid at runtime):
|
||||
|
||||
::
|
||||
|
||||
def foo(*args: Tuple[*Ts]): ...
|
||||
|
||||
def bar(x: Tuple[int, ...]):
|
||||
foo(*x) # NOT valid
|
||||
|
||||
Finally, note that a type variable tuple may *not* be used as the type of
|
||||
``**kwargs``. (We do not yet know of a use-case for this feature, so prefer
|
||||
``**kwargs``. (We do not yet know of a use case for this feature, so we prefer
|
||||
to leave the ground fresh for a potential future PEP.)
|
||||
|
||||
::
|
||||
|
||||
# NOT valid
|
||||
def foo(**kwargs: Ts): ...
|
||||
def foo(**kwargs: *Ts): ...
|
||||
|
||||
Type Variable Tuples with ``Callable``
|
||||
''''''''''''''''''''''''''''''''''''''
|
||||
--------------------------------------
|
||||
|
||||
Type variable tuples can also be used in the arguments section of a
|
||||
``Callable``:
|
||||
|
|
@ -286,116 +389,98 @@ Type variable tuples can also be used in the arguments section of a
|
|||
::
|
||||
|
||||
class Process:
|
||||
def __init__(target: Callable[[*Ts], Any], args: Tuple[*Ts]): ...
|
||||
def __init__(
|
||||
target: Callable[[*Ts], Any],
|
||||
args: Tuple[*Ts]
|
||||
): ...
|
||||
|
||||
def func(arg1: int, arg2: str): ...
|
||||
|
||||
Process(target=func, args=(0, 'foo')) # Passes type-check
|
||||
Process(target=func, args=('foo', 0)) # Fails type-check
|
||||
Process(target=func, args=(0, 'foo')) # Valid
|
||||
Process(target=func, args=('foo', 0)) # Error
|
||||
|
||||
However, note that as of this PEP, if a type variable tuple does appear in
|
||||
the arguments section of a ``Callable``, it must appear alone.
|
||||
That is, `Type Prefixing` is not supported in the context of ``Callable``.
|
||||
(Use cases where this might otherwise be desirable are likely covered through use
|
||||
of either a `ParamSpec` from PEP 612, or a type variable tuple in the `__call__`
|
||||
signature of a callback protocol from PEP 544.)
|
||||
|
||||
Type Variable Tuples with ``Union``
|
||||
'''''''''''''''''''''''''''''''''''
|
||||
-----------------------------------
|
||||
|
||||
Finally, type variable tuples can be used with ``Union``:
|
||||
Type variable tuples can also be used with ``Union``:
|
||||
|
||||
::
|
||||
|
||||
def f(*args: *Ts) -> Union[*Ts]:
|
||||
def f(*args: Tuple[*Ts]) -> Union[*Ts]:
|
||||
return random.choice(args)
|
||||
|
||||
f(1, 'foo') # Inferred type is Union[int, str]
|
||||
|
||||
If the type variable tuple is empty (e.g. if we had ``*args: *Ts``
|
||||
More than one type variable tuple may appear in the the parameter list
|
||||
to ``Union``:
|
||||
|
||||
::
|
||||
|
||||
def cond_random_choice(
|
||||
cond: bool,
|
||||
cond_true: Tuple[*Ts1],
|
||||
cond_false: Tuple[*Ts2]
|
||||
) -> Union[*Ts1, *Ts2]:
|
||||
if cond:
|
||||
return random.choice(cond_true)
|
||||
else:
|
||||
return random.choice(cond_false)
|
||||
|
||||
# Inferred type is Union[int, str, float]
|
||||
cond_random_choice(True, (1, 'foo'), (0.0, 'bar'))
|
||||
|
||||
If the type variable tuple is empty (e.g. if we had ``*args: Tuple[*Ts]``
|
||||
and didn't pass any arguments), the type checker should
|
||||
raise an error on the ``Union`` (matching the behaviour of ``Union``
|
||||
at runtime, which requires at least one type argument).
|
||||
|
||||
``Map``
|
||||
Aliases
|
||||
-------
|
||||
|
||||
To enable typing of functions such as ``map`` and ``zip``, we introduce the
|
||||
``Map`` type operator. Not to be confused with the existing operator
|
||||
``typing.Mapping``, ``Map`` is analogous to ``map``, but for types:
|
||||
Generic aliases can be created using a type variable tuple in
|
||||
a similar way to regular type variables:
|
||||
|
||||
::
|
||||
|
||||
from typing import Map
|
||||
IntTuple = Tuple[int, *Ts]
|
||||
IntTuple[float, bool] # Equivalent to Tuple[int, float, bool]
|
||||
|
||||
def args_to_lists(*args: *Ts) -> Map[List, Ts]: ...
|
||||
|
||||
args_to_lists(1, 'a') # Inferred type is Tuple[List[int], List[str]]
|
||||
|
||||
``Map`` takes two operands. The first operand is a parameterizable
|
||||
type (or type alias [#type_aliases]_) such as ``Tuple``, ``List``, or a
|
||||
user-defined generic class. The second operand is a type variable tuple.
|
||||
The result of ``Map`` is a ``Tuple``, where the Nth type in the ``Tuple`` is
|
||||
the first operand parameterized by the Nth type in the type variable tuple.
|
||||
|
||||
Because ``Map`` returns a parameterized ``Tuple``, it can be used anywhere
|
||||
that a type variable tuple would be. For example, as the type of ``*args``:
|
||||
|
||||
::
|
||||
|
||||
# Equivalent to 'arg1: List[T1], arg2: List[T2], ...'
|
||||
def foo(*args: *Map[List, Ts]): ...
|
||||
# Ts is bound to Tuple[int, str]
|
||||
foo([1], ['a'])
|
||||
|
||||
As a return type:
|
||||
As this example shows, all type parameters passed to the alias are
|
||||
bound to the type variable tuple. If no type parameters are given,
|
||||
or if an explicitly empty list of type parameters are given,
|
||||
type variable tuple in the alias is simply ignored:
|
||||
|
||||
::
|
||||
|
||||
# Equivalent to '-> Tuple[List[T1], List[T2], ...]'
|
||||
def bar(*args: *Ts) -> Map[List, Ts]: ...
|
||||
# Ts is bound to Tuple[float, bool]
|
||||
# Inferred type is Tuple[List[float], List[bool]]
|
||||
bar(1.0, True)
|
||||
# Both equivalent to Tuple[int]
|
||||
IntTuple
|
||||
IntTuple[()]
|
||||
|
||||
And as an argument type:
|
||||
Normal ``TypeVar`` instances can also be used in such aliases:
|
||||
|
||||
::
|
||||
|
||||
# Equivalent to 'arg: Tuple[List[T1], List[T2], ...]'
|
||||
def baz(arg: Map[List, Ts]): ...
|
||||
# Ts is bound to Tuple[bool, bool]
|
||||
baz(([True], [False]))
|
||||
T = TypeVar('T')
|
||||
Foo = Tuple[T, *Ts]
|
||||
|
||||
``map`` and ``zip``
|
||||
'''''''''''''''''''
|
||||
# T is bound to `int`; Ts is bound to `bool, str`
|
||||
Foo[int, bool, str]
|
||||
|
||||
``Map`` allows us to specify the signature of ``map`` as:
|
||||
|
||||
::
|
||||
|
||||
Ts = TypeVarTuple('Ts')
|
||||
R = TypeVar(R)
|
||||
|
||||
def map(func: Callable[[*Ts], R],
|
||||
*iterables: *Map[Iterable, Ts]) -> Iterable[R]: ...
|
||||
|
||||
def func(int, str) -> float: ...
|
||||
# Ts is bound to Tuple[int, str]
|
||||
# Map[Iterable, Ts] is Iterable[int], Iterable[str]
|
||||
# Therefore, iter1 must be type Iterable[int],
|
||||
# and iter2 must be type Iterable[str]
|
||||
map(func, iter1, iter2)
|
||||
|
||||
Similarly, we can specify the signature of ``zip`` as:
|
||||
|
||||
::
|
||||
|
||||
def zip(*iterables: *Map[Iterable, Ts]) -> Iterator[Ts]): ...
|
||||
|
||||
l1: List[int]
|
||||
l2: List[str]
|
||||
zip(l1, l2) # Iterator[Tuple[int, str]]
|
||||
Note that the same rules for `Type Prefixing` apply for aliases.
|
||||
In particular, only one ``TypeVarTuple`` may occur within an alias,
|
||||
and the ``TypeVarTuple`` must be at the end of the alias.
|
||||
|
||||
Overloads for Accessing Individual Types
|
||||
----------------------------------------
|
||||
|
||||
``Map`` allows us to operate on types in a bulk fashion. For situations where we
|
||||
require access to each individual type, overloads can be used with individual
|
||||
``TypeVar`` instances in place of the type variable tuple:
|
||||
For situations where we require access to each individual type, overloads can be used with individual ``TypeVar`` instances in place of the type variable tuple:
|
||||
|
||||
::
|
||||
|
||||
|
|
@ -404,7 +489,7 @@ require access to each individual type, overloads can be used with individual
|
|||
Axis2 = TypeVar('Axis2')
|
||||
Axis3 = TypeVar('Axis3')
|
||||
|
||||
class Array(Generic[*Shape]): ...
|
||||
class Array(Generic[Shape]): ...
|
||||
|
||||
@overload
|
||||
def transpose(
|
||||
|
|
@ -421,232 +506,6 @@ overloads for each possible rank is, of course, a rather cumbersome
|
|||
solution. However, it's the best we can do without additional type
|
||||
manipulation mechanisms, which are beyond the scope of this PEP.)
|
||||
|
||||
Concatenating Other Types to a Type Variable Tuple
|
||||
--------------------------------------------------
|
||||
|
||||
If an unpacked type variable tuple appears with other types in the same type parameter
|
||||
list, the effect is to concatenate those types with the types in the type variable
|
||||
tuple. For example, concatenation in a function return type:
|
||||
|
||||
::
|
||||
|
||||
Batch = NewType('int')
|
||||
Height = NewType('int')
|
||||
Width = NewType('int')
|
||||
|
||||
class Array(Generic[*Shape]): ...
|
||||
|
||||
def add_batch(x: Array[*Shape]) -> Array[Batch, *Shape]: ...
|
||||
|
||||
x: Array[Height, Width]
|
||||
y = add_batch(x) # Inferred type is Array[Batch, Height, Width]
|
||||
|
||||
In function argument types:
|
||||
|
||||
::
|
||||
|
||||
def batch_sum(x: Array[Batch, *Shape]) -> Array[*Shape]: ...
|
||||
|
||||
x: Array[Batch, Height, Width]
|
||||
y = batch_sum(x) # Inferred type is Array[Height, Width]
|
||||
|
||||
And in class type parameters:
|
||||
|
||||
::
|
||||
|
||||
class BatchArray(Generic[Batch, *Shape]):
|
||||
def sum(self) -> Array[*Shape]: ...
|
||||
|
||||
x: BatchArray[Batch, Height, Width]
|
||||
y = x.sum() # Inferred type is Array[Height, Width]
|
||||
|
||||
Concatenation can involve both prefixing and suffixing, and
|
||||
can include an arbitrary number of types:
|
||||
|
||||
::
|
||||
|
||||
def foo(x: Tuple[*Ts]) -> Tuple[int, str, *Ts, bool]: ...
|
||||
|
||||
It is also possible to concatenate type variable tuples with regular
|
||||
type variables:
|
||||
|
||||
::
|
||||
|
||||
T = TypeVar('T')
|
||||
|
||||
def first_axis_sum(x: Array[T, *Shape]) -> Array[*Shape]: ...
|
||||
|
||||
x: Array[Time, Height, Width]
|
||||
y = first_axis_sum(x) # Inferred type is Array[Height, Width]
|
||||
|
||||
Finally, concatenation can also occur in the argument list to ``Callable``:
|
||||
|
||||
::
|
||||
|
||||
def f(func: Callable[[int, *Ts], Any]) -> Tuple[*Ts]: ...
|
||||
|
||||
def foo(int, str, float): ...
|
||||
def bar(str, int, float): ...
|
||||
|
||||
f(foo) # Valid; inferred type is Tuple[str, float]
|
||||
f(bar) # Not valid
|
||||
|
||||
And in ``Union``:
|
||||
|
||||
::
|
||||
|
||||
def f(*args: *Ts) -> Union[*Ts, float]: ...
|
||||
|
||||
f(0, 'spam') # Inferred type is Union[int, str, float]
|
||||
|
||||
Concatenating Multiple Type Variable Tuples
|
||||
-------------------------------------------
|
||||
|
||||
We can also concatenate *multiple* type variable tuples, but only in cases
|
||||
where the types bound to each type variable tuple can be inferred
|
||||
unambiguously. Note that this is not always the case:
|
||||
|
||||
::
|
||||
|
||||
# Type checker should raise an error on definition of func;
|
||||
# how would we know which types are bound to Ts1, and which
|
||||
# are bound to Ts2?
|
||||
def func(ham: Tuple[*Ts1, *Ts2]): ...
|
||||
|
||||
# Ts1 = Tuple[int, str], Ts2 = Tuple[bool]?
|
||||
# Or Ts1 = Tuple[int], Ts2 = Tuple[str, bool]?
|
||||
ham: Tuple[int, str, bool]
|
||||
func(ham)
|
||||
|
||||
In general, some kind of extra constraint is necessary in order
|
||||
for the ambiguity to be resolved. This is usually provided by
|
||||
an un-concatenated usage of the type variable tuple elsewhere in
|
||||
the same signature.
|
||||
|
||||
For example, resolving ambiguity in an argument:
|
||||
|
||||
::
|
||||
|
||||
def func(ham: Tuple[*Ts1, *Ts2], spam: Ts2): ...
|
||||
|
||||
# Ts1 is bound to Tuple[int], Ts2 to Tuple[str, bool]
|
||||
ham: Tuple[int, str, bool]
|
||||
spam: Tuple[str, bool]
|
||||
func(ham, spam)
|
||||
|
||||
In a return type:
|
||||
|
||||
::
|
||||
|
||||
def func(ham: Ts1, spam: Ts2) -> Tuple[*Ts1, *Ts2]): ...
|
||||
|
||||
ham: Tuple[int]
|
||||
spam: Tuple[str, bool]
|
||||
# Return type is Tuple[int, str, bool]
|
||||
func(ham, spam)
|
||||
|
||||
Note, however, that the same cannot be done with generic classes:
|
||||
|
||||
::
|
||||
|
||||
# No way to add extra constraints about Ts1 and Ts2,
|
||||
# so this is not valid
|
||||
class C(Generic[*Ts1, *Ts2]): ...
|
||||
|
||||
Generics in Multiple Type Variable Tuples
|
||||
-----------------------------------------
|
||||
|
||||
If we *do* wish to use multiple type variable tuples in a type signature
|
||||
that would otherwise not resolve the ambiguity, it is also possible
|
||||
to make the type bindings explicit by using a type variable tuple directly,
|
||||
without unpacking it. When then instantiating, for example, the class in
|
||||
question, the types corresponding to each type variable tuple must
|
||||
be wrapped in a ``Tuple``:
|
||||
|
||||
::
|
||||
|
||||
class C(Generic[Ts1, Ts2]): ...
|
||||
|
||||
# Ts1 = Tuple[int, str]
|
||||
# Ts2 = Tuple[bool]
|
||||
c: C[Tuple[int, str], Tuple[bool]] = C()
|
||||
|
||||
Similarly for functions:
|
||||
|
||||
::
|
||||
|
||||
def foo(x: Tuple[Ts1, Ts2]): ...
|
||||
|
||||
# Ts1 = Tuple[int, float]
|
||||
# Ts2 = Tuple[bool]
|
||||
x: Tuple[Tuple[int, float], Tuple[bool]]
|
||||
foo(x)
|
||||
|
||||
Aliases
|
||||
-------
|
||||
|
||||
Generic aliases can be created using a type variable tuple in
|
||||
a similar way to regular type variables:
|
||||
|
||||
::
|
||||
|
||||
IntTuple = Tuple[int, *Ts]
|
||||
IntTuple[float, bool] # Equivalent to Tuple[int, float, bool]
|
||||
|
||||
As this example shows, all type arguments passed to the alias are
|
||||
bound to the type variable tuple. If no type arguments are given,
|
||||
the type variable tuple holds no types:
|
||||
|
||||
::
|
||||
|
||||
IntTuple # Equivalent to Tuple[int]
|
||||
|
||||
Type variable tuples can also be used without unpacking:
|
||||
|
||||
::
|
||||
|
||||
IntTuple = Tuple[int, Ts]
|
||||
IntTuple[float, bool] # Equivalent to Tuple[int, Tuple[float, bool]]
|
||||
IntTuple # Tuple[int, Tuple[]]
|
||||
|
||||
At most a single distinct type variable tuple can occur in an alias:
|
||||
|
||||
::
|
||||
|
||||
# Invalid
|
||||
Foo = Tuple[Ts1, int, Ts2]
|
||||
# Why? Because there would be no way to decide which types should
|
||||
# be bound to which type variable tuple:
|
||||
Foo[float, bool, str]
|
||||
# Equivalent to Tuple[float, bool, int, str]?
|
||||
# Or Tuple[float, int, bool, str]?
|
||||
|
||||
The same type variable tuple may be used multiple times, however:
|
||||
|
||||
::
|
||||
|
||||
Bar = Tuple[*Ts, *Ts]
|
||||
Bar[int, float] # Equivalent to Tuple[int, float, int, float]
|
||||
|
||||
Finally, type variable tuples can be used in combination with
|
||||
normal type variables. In this case, the number of type arguments must
|
||||
be equal to or greater than the number of distinct normal type variables:
|
||||
|
||||
::
|
||||
|
||||
Baz = Tuple[T1, *Ts, T2, T1]
|
||||
|
||||
# T1 bound to int, T2 bound to bool, Ts empty
|
||||
# Equivalent to Tuple[int, bool, int]
|
||||
Baz[int, bool]
|
||||
|
||||
# T1 bound to int
|
||||
# Ts bound to Tuple[float, bool]
|
||||
# T2 bound to str
|
||||
# So equivalent to Tuple[int, float, bool, str, int]
|
||||
Baz[int, float, bool, str]
|
||||
|
||||
|
||||
An Ideal Array Type: One Possible Example
|
||||
=========================================
|
||||
|
||||
|
|
@ -693,6 +552,11 @@ in a complete array type.
|
|||
# E.g. Array1D[np.uint8]
|
||||
Array1D = Array[DataType, Ndim[Literal[1]]]
|
||||
|
||||
Final Notes
|
||||
===========
|
||||
|
||||
**Slice expressions**: type variable tuples may *not* appear in slice expressions.
|
||||
|
||||
Rationale and Rejected Ideas
|
||||
============================
|
||||
|
||||
|
|
@ -710,40 +574,8 @@ by simply defining aliases for each possible number of type parameters:
|
|||
However, this seems somewhat clumsy - it requires users to unnecessarily
|
||||
pepper their code with 1s, 2s, and so on for each rank necessary.
|
||||
|
||||
Naming of ``Map``
|
||||
-----------------
|
||||
|
||||
One downside to the name ``Map`` is that it might suggest a hash map. We
|
||||
considered a number of different options for the name of this operator.
|
||||
|
||||
* ``ForEach``. This is rather long, and we thought might imply a side-effect.
|
||||
* ``Transform``. The meaning of this isn't obvious enough at first glance.
|
||||
* ``Apply``. This is inconsistent with ``apply``, an older Python function
|
||||
which enabled conversion of iterables to arguments before the star
|
||||
operator was introduced.
|
||||
|
||||
In the end, we decided that ``Map`` was good enough.
|
||||
|
||||
Nesting ``Map``
|
||||
---------------
|
||||
|
||||
Since the result of ``Map`` is a parameterised ``Tuple``, it should be
|
||||
possible to use the output of a ``Map`` as the input to another ``Map``:
|
||||
|
||||
::
|
||||
|
||||
Map[Tuple, Map[List, Ts]]
|
||||
|
||||
If ``Ts`` here were bound to ``Tuple[int, str]``, the result of the
|
||||
inner ``Map`` would be ``Tuple[List[int], List[str]]``, so the result
|
||||
of the outer map would be ``Tuple[Tuple[List[int]], Tuple[List[str]]]``.
|
||||
|
||||
We chose not to highlight this fact because of a) how confusing it is,
|
||||
and b) lack of a specific use-case. Whether to support nested ``Map``
|
||||
is left to the implementation.
|
||||
|
||||
Naming of ``TypeVarTuple``
|
||||
--------------------------
|
||||
Construction of ``TypeVarTuple``
|
||||
--------------------------------
|
||||
|
||||
``TypeVarTuple`` began as ``ListVariadic``, based on its naming in
|
||||
an early implementation in Pyre.
|
||||
|
|
@ -752,12 +584,15 @@ We then changed this to ``TypeVar(list=True)``, on the basis that a)
|
|||
it better emphasises the similarity to ``TypeVar``, and b) the meaning
|
||||
of 'list' is more easily understood than the jargon of 'variadic'.
|
||||
|
||||
We finally settled on ``TypeVarTuple`` based on the justification
|
||||
that c) this emphasises the tuple-like behaviour, and d) type variable
|
||||
tuples are a sufficiently different kind of thing to regular
|
||||
type variables that we may later wish to support keyword arguments
|
||||
to its constructor that should not be supported by regular
|
||||
type variables (such as ``arbitrary_len`` [#arbitrary_len]_).
|
||||
Once we'd decided that a variadic type variable should behave like a ``Tuple``,
|
||||
we also considered ``TypeVar(bound=Tuple)``, which is similarly intuitive
|
||||
and accomplishes most what we wanted without requiring any new arguments to
|
||||
``TypeVar``. However, we realised this may constrain us in the future, if
|
||||
for example we want type bounds or variance to function slightly differently
|
||||
for variadic type variables than what the semantics of ``TypeVar`` might
|
||||
otherwise imply. Also, we may later wish to support arguments that should not be supported by regular type variables (such as ``arbitrary_len`` [#arbitrary_len]_).
|
||||
|
||||
We therefore settled on ``TypeVarTuple``.
|
||||
|
||||
Backwards Compatibility
|
||||
=======================
|
||||
|
|
@ -776,9 +611,6 @@ TODO
|
|||
Footnotes
|
||||
==========
|
||||
|
||||
.. [#hkt] A third potential use is in enabling higher-kinded types that take
|
||||
an arbitrary number of type operands, but we do not discuss this use
|
||||
here.
|
||||
|
||||
.. [#batch] 'Batch' is machine learning parlance for 'a number of'.
|
||||
|
||||
|
|
@ -791,11 +623,6 @@ Footnotes
|
|||
shape begins with 'time × batch', then ``videos_batch[1][0]`` would select the
|
||||
same frame.
|
||||
|
||||
.. [#type_aliases] For example, in ``asyncio`` [#asyncio]_, it is convenient
|
||||
to define a type alias
|
||||
``_FutureT = Union[Future[_T], Generator[Any, None, _T], Awaitable[_T]]``.
|
||||
We should be able to apply ``Map`` to such aliases - e.g. ``Map[_FutureT, Ts]``.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
|
|
@ -810,10 +637,6 @@ References
|
|||
.. [#syntax-proposal] Shape annotation syntax proposal:
|
||||
https://docs.google.com/document/d/1But-hjet8-djv519HEKvBN6Ik2lW3yu0ojZo6pG9osY/edit
|
||||
|
||||
.. [#zip-sig] ``typeshed/builtins.pyi``: https://github.com/python/typeshed/blob/27dfbf68aaffab4f1ded7dc1b96f6f82f536a09d/stdlib/2and3/builtins.pyi#L1710-L1733
|
||||
|
||||
.. [#asyncio] ``typeshed/asyncio/tasks.pyi``: https://github.com/python/typeshed/blob/193c7cb93283ad4ca2a65df74c565e56bfe72b7e/stdlib/3/asyncio/tasks.pyi#L45-L154
|
||||
|
||||
.. [#arbitrary_len] Discussion on Python typing-sig mailing list: https://mail.python.org/archives/list/typing-sig@python.org/thread/SQVTQYWIOI4TIO7NNBTFFWFMSMS2TA4J/
|
||||
|
||||
|
||||
|
|
@ -825,8 +648,7 @@ Thank you to **Alfonso Castaño**, **Antoine Pitrou**, **Bas v.B.**, **David Fos
|
|||
**Sergei Lebedev** and **Vladimir Mikulik** for helpful feedback and suggestions on
|
||||
drafts of this PEP.
|
||||
|
||||
Thank you especially to **Lucio**, for suggesting the star syntax, which has made
|
||||
multiple aspects of this proposal much more concise and intuitive.
|
||||
Thank you especially to **Lucio**, for suggesting the star syntax, which has made multiple aspects of this proposal much more concise and intuitive.
|
||||
|
||||
Resources
|
||||
=========
|
||||
|
|
@ -865,4 +687,3 @@ CC0-1.0-Universal license, whichever is more permissive.
|
|||
fill-column: 70
|
||||
coding: utf-8
|
||||
End:
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue