PEP 646: Update draft (#1856)

This is the more-or-less final version now. I'll do one last read-through to confirm everything's consistent, then post in typing-sig to confirm it's ready for the Steering Committee.

Semantic changes:
* Remove support for type variable tuples in `Union`. Apparently the implementation would be tricky in Pyre because of special-casing around `Union`. That could be evidence that it would be tricky in Mypy and pytype. Since we don't have a specific use-case in mind, I think this is fine.
* Support concatenation in `Callable`. Pradeep pointed ou that `ParamSpec` doesn't cover all potential use-cases because it doesn't support concatenating suffixes (in turn because `ParamSpec`s can contain keyword arguments, and concatenation is positional).

Readability changes:
* Remove the section on a full `Array` example. We've moved the important thing here - an explicit confirmation that `Array` can be generic in both datatype and shape - to the beginning of the PEP to make it more obvious. We've removed the `Ndim` thing after someone pointed out that e.g. `Ndim[Literal[2]]` would be incompatible with `Shape[Any, Any]`; I just don't think it's going to work. All that's left is the aliases, which we've moved into the Aliases section itself.

Other changes:
* Fill out Backwards Compatibility section.
* Reference initial CPython implementation.
This commit is contained in:
Matthew Rahtz 2021-03-03 00:33:15 +00:00 committed by GitHub
parent a3f0183936
commit b95239fef1
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 71 additions and 92 deletions

View File

@ -476,38 +476,14 @@ Type variable tuples can also be used in the arguments section of a
Process(target=func, args=(0, 'foo')) # Valid
Process(target=func, args=('foo', 0)) # Error
However, note that as of this PEP, if a type variable tuple does appear in
the arguments section of a ``Callable``, it must appear alone.
That is, `Type Concatenation`_ is not supported in the context of ``Callable``.
(Use cases where this might otherwise be desirable are likely covered through use
of either a ``ParamSpec`` from PEP 612, or a type variable tuple in the ``__call__``
signature of a callback protocol from PEP 544.)
Type Variable Tuples with ``Union``
-----------------------------------
Type variable tuples can also be used with ``Union``:
Other types and normal type variables can also be prefixed/suffixed
to the type variable tuple:
::
def f(*args: *Ts) -> Union[*Ts]:
return random.choice(args)
T = TypeVar('T')
f(1, 'foo') # Inferred type is Union[int, str]
Here, if the type variable tuple is empty (e.g. if we had ``*args: *Ts``
and didn't pass any arguments), the type checker should
raise an error on the ``Union`` (matching the behaviour of ``Union``
at runtime, which requires at least one type argument).
Other types can also be included in the ``Union``:
::
def f(*args :*Ts) -> Union[int, str, *Ts]: ...
However, note that as elsewhere, only a single type variable tuple
may occur within a ``Union``.
def foo(f: Callable[[int, *Ts, T], Tuple[T, *Ts]]): ...
Aliases
-------
@ -518,32 +494,64 @@ a similar way to regular type variables:
::
IntTuple = Tuple[int, *Ts]
NamedArray = Tuple[str, Array[*Ts]]
IntTuple[float, bool] # Equivalent to Tuple[int, float, bool]
NamedArray[Height] # Equivalent to Tuple[str, Array[Height]]
As this example shows, all type parameters passed to the alias are
bound to the type variable tuple. If no type parameters are given,
or if an explicitly empty list of type parameters are given,
type variable tuple in the alias is simply ignored:
bound to the type variable tuple.
Importantly for our ``Array`` example (see `Summary Examples`_), this
allows us to define convenience aliases for arrays of a fixed shape
or datatype:
::
# Both equivalent to Tuple[int]
IntTuple
IntTuple[()]
Shape = TypeVarTuple('Shape')
DType = TypeVar('DType')
class Array(Generic[DType, *Shape]):
# E.g. Float32Array[Height, Width, Channels]
Float32Array = Array[np.float32, *Shape]
# E.g. Array1D[np.uint8]
Array1D = Array[DType, Any]
If an explicitly empty type parameter list is given, the type variable
tuple in the alias is set empty:
::
IntTuple[()] # Equivalent to Tuple[int]
NamedArray[()] # Equivalent to Tuple[str, Array[()]]
If the type parameter list is omitted entirely, the alias is
compatible with arbitrary type parameters:
::
def takes_float_array_of_any_shape(x: Float32Array): ...
x: Float32Array[Height, Width] = Array()
takes_float_array_of_any_shape(x) # Valid
def takes_float_array_with_specific_shape(y: Float32Array[Height, Width]): ...
y: Float32Array = Array()
takes_float_array_with_specific_shape(y) # Valid
Normal ``TypeVar`` instances can also be used in such aliases:
::
T = TypeVar('T')
Foo = Tuple[T, *Ts]
Foo = Tuple[*Ts, T]
# T is bound to `int`; Ts is bound to `bool, str`
Foo[int, bool, str]
Note that the same rules for `Type Concatenation`_ apply for aliases.
In particular, only one ``TypeVarTuple`` may occur within an alias,
and the ``TypeVarTuple`` must be at the end of the alias.
# Ts bound to Tuple[int], T to int
Foo[str, int]
# Ts bound to Tuple[()], T to int
Foo[int]
# T bound to Any, Ts to an arbitrary number of Any
Foo
Overloads for Accessing Individual Types
----------------------------------------
@ -574,51 +582,6 @@ overloads for each possible rank is, of course, a rather cumbersome
solution. However, it's the best we can do without additional type
manipulation mechanisms, which are beyond the scope of this PEP.)
An Ideal Array Type: One Possible Example
=========================================
Type variable tuples allow us to make significant progress on the
typing of arrays. However, the array class we have sketched
out in this PEP is still missing some desirable features. [#typing-ideas]_
The most crucial feature missing is the ability to specify
the data type (e.g. ``np.float32`` or ``np.uint8``). This is important
because some numerical computing libraries will silently cast
types, which can easily lead to hard-to-diagnose bugs.
Additionally, it might be useful to be able to specify the rank
instead of the full shape. This could be useful for cases where
axes don't have obvious semantic meaning like 'height' or 'width',
or where the array is very high-dimensional and writing out all
the axes would be too verbose.
Here is one possible example of how these features might be implemented
in a complete array type.
::
# E.g. Ndim[Literal[3]]
Integer = TypeVar('Integer')
class Ndim(Generic[Integer]): ...
# E.g. Shape[Height, Width]
# (Where Height and Width are custom types)
Axes = TypeVarTuple('Axes')
class Shape(Generic[*Axes]): ...
DataType = TypeVar('DataType')
ShapeType = TypeVar('ShapeType', Ndim, Shape)
# The most verbose type
# E.g. Array[np.float32, Ndim[Literal[3]]
# Array[np.uint8, Shape[Height, Width, Channels]]
class Array(Generic[DataType, ShapeType]): ...
# Type aliases for less verbosity
# E.g. Float32Array[Height, Width, Channels]
Float32Array = Array[np.float32, Shape[*Axes]]
# E.g. Array1D[np.uint8]
Array1D = Array[DataType, Ndim[Literal[1]]]
Rationale and Rejected Ideas
============================
@ -710,20 +673,32 @@ in how much of their code they wish to annotate, and to enable compatibility
between old unannotated code and new versions of libraries which do use
these type annotations.
Backwards Compatibility
=======================
TODO
The ``Unpack`` version of the PEP should be back-portable to previous
versions of Python.
* ``Tuple`` needs to be upgraded to support parameterization with a
type variable tuple.
Gradual typing is enabled by the fact that unparameterised variadic classes
are compatible with an arbitrary number of type parameters. This means
that if existing classes are made generic, a) all existing (unparameterised)
uses of the class will still work, and b) parameterised and unparameterised
versions of the class can be used together (relevant if, for example, library
code is updated to use parameters while user code is not, or vice-versa).
Reference Implementation
========================
Two reference implementations exist: one in Pyre, as of TODO, and one in
Pyright, as of v1.1.108.
Two reference implementations of type-checking functionality exist:
one in Pyre, as of TODO, and one in Pyright, as of v1.1.108.
A preliminary implementation of the ``Unpack`` version of the PEP in CPython
is available in `cpython/23527`_. A preliminary version of the version
using the star operator, based on an early implementation of PEP 637,
is also available at `mrahtz/cpython/pep637+646`_.
Footnotes
==========
@ -758,6 +733,10 @@ References
.. [#arbitrary_len] Discussion on Python typing-sig mailing list: https://mail.python.org/archives/list/typing-sig@python.org/thread/SQVTQYWIOI4TIO7NNBTFFWFMSMS2TA4J/
.. _cpython/23527: https://github.com/python/cpython/pull/24527
.. _mrahtz/cpython/pep637+646: https://github.com/mrahtz/cpython/tree/pep637%2B646
Acknowledgements
================