PEP 646: various improvements (#1825)

Semantic changes:
* Unparameterised variadics are compatible with parameterised variadics in both direction
* Allow suffixing

Readability changes:
* De-emphasise unparameterised `Array` behaving like `Array[Any, ...]` to avoid the suggestion that `Array[Any, ...]` might be supported
* Explicitly forbid `Array[Any, ...]` and similar
* Clarify that only `*args` can be annotated as `*Ts`; other arguments must use `Tuple[*Ts]`
* Remove 'Final Notes' section (I was hoping we'd have more edge cases to add to this, but in the end there haven't been any, and the only example we do have is obscure enough that I think we can just remove it)

Also some minor typo fixes.

pep-0646.rst

@@ -70,7 +70,7 @@ generic in an arbitrary number of axes. One way around this would be to use a di
 ...but this would be cumbersome, both for users (who would have to sprinkle 1s and 2s
 and so on throughout their code) and for the authors of array libraries (who would have to duplicate implementations throughout multiple classes).
 
-Variadic generics are necessary for a ``Array`` that is generic in an arbitrary
+Variadic generics are necessary for an ``Array`` that is generic in an arbitrary
 number of axes to be cleanly defined as a single class.
 
 Specification
@@ -205,24 +205,36 @@ any type parameters, it behaves as if its type parameters are ``Any, ...``
 
 ::
 
-    def takes_array_of_any_rank(arr: Array): ...
+    def takes_any_array(arr: Array): ...
 
     x: Array[Height, Width]
-    takes_array_of_any_rank(x)  # Valid
+    takes_any_array(x)  # Valid
     y: Array[Time, Height, Width]
-    takes_array_of_any_rank(y)  # Also valid
+    takes_any_array(y)  # Also valid
 
-This enables gradual typing: existing functions with arguments annotated as being,
-for example, a plain ``tf.Tensor``, will still be valid even if called with
+This enables gradual typing: existing functions accepting, for example,
+a plain ``tf.Tensor`` will still be valid even if called with
 a parameterised ``Tensor[Height, Width]``.
 
+This also works in the opposite direction:
+
+::
+
+    def takes_specific_array(arr: Array[Height, Width]): ...
+
+    z: Array
+    takes_specific_array(z)
+
+Thereby even if libraries are updated to use types like ``Array[Height, Width]``,
+users of those libraries won't be forced to also apply type annotations to
+all of their code; users still have a choice about what parts of their code
+to type and which parts to not.
+
 Type Variable Tuples Must Have Known Length
 '''''''''''''''''''''''''''''''''''''''''''
 
-Note that in the ``takes_array_of_any_rank`` example in the previous section,
-``Array`` behaved as if it were ``Tuple[int, ...]``. This situation - when
-type parameters are not specified - is the *only* case when a type variable
-tuple may be bound to an unknown-length type. That is:
+Type variables tuples may not be bound to a type with unknown length.
+That is:
 
 ::
 
@@ -231,13 +243,25 @@ tuple may be bound to an unknown-length type. That is:
     x: Tuple[float, ...]
     foo(x)  # NOT valid; Ts would be bound to ``Tuple[float, ...]``
 
-(If this is confusing - didn't we say that type variable tuples are a stand-in
+If this is confusing - didn't we say that type variable tuples are a stand-in
 for an *arbitrary* number of types? - note the difference between the
 length of the type variable tuple *itself*, and the length of the type it is
 *bound* to. Type variable tuples themselves can be of arbitrary length -
 that is, they can be bound to ``Tuple[int]``, ``Tuple[int, int]``, and
-so on - but the length of the types they are bound to must be of known length -
-that is, ``Tuple[int, int]``, but not ``Tuple[int, ...]``.)
+so on - but the types they are bound to must be of known length -
+that is, ``Tuple[int, int]``, but not ``Tuple[int, ...]``.
+
+Note that, as a result of this rule, omitting the type parameter list is the
+*only* way of instantiating a generic type with an arbitrary number of
+type parameters. For example, ``Array`` may *behave* like ``Array[Any, ...]``,
+but it cannot be instantiated using ``Array[Any, ...]``, because this would
+bind its type variable tuple to ``Tuple[Any, ...]``:
+
+::
+
+    x: Array            # Valid
+    y: Array[int, ...]  # Error
+    z: Array[Any, ...]  # Error
 
 Type Variable Tuple Equality
 ''''''''''''''''''''''''''''
@@ -256,7 +280,8 @@ a ``Tuple`` of a ``Union`` of types:
 
 We do *not* allow this; type unions may *not* appear within the ``Tuple``.
 If a type variable tuple appears in multiple places in a signature,
-the types must match exactly:
+the types must match exactly (the list of type parameters must be the same
+length, and the type parameters themselves must be identical):
 
 ::
 
@@ -283,11 +308,11 @@ As of this PEP, only a single type variable tuple may appear in a type parameter
 
 (``Union`` is the one exception to this rule; see `Type Variable Tuples with Union`_.)
 
-Type Prefixing
---------------
+Type Concatenation
+------------------
 
 Type variable tuples don't have to be alone; normal types can be
-prefixed to them:
+prefixed and/or suffixed:
 
 ::
 
@@ -296,12 +321,15 @@ prefixed to them:
 
     def add_batch_axis(x: Array[*Shape]) -> Array[Batch, *Shape]: ...
     def del_batch_axis(x: Array[Batch, *Shape]) -> Array[*Shape]: ...
+    def add_batch_channels(x: Array[*Shape]) -> Array[Batch, *Shape, Channels]: ...
 
-    x: Array[Height, Width]
-    y = add_batch(x)  # Inferred type is Array[Batch, Height, Width]
-    z = del_batch(y)  # Array[Height, Width]
+    a: Array[Height, Width]
+    b = add_batch(a)           # Inferred type is Array[Batch, Height, Width]
+    c = del_batch(b)           # Array[Height, Width]
+    d = add_batch_channels(a)  # Array[Batch, Height, Width, Channels]
     
-Normal ``TypeVar`` instances can also be prefixed:
+
+Normal ``TypeVar`` instances can also be prefixed and/or suffixed:
 
 ::
 
@@ -316,8 +344,6 @@ Normal ``TypeVar`` instances can also be prefixed:
     z = prefix_tuple(x=0, y=(True, 'a'))
     # Inferred type of z is Tuple[int, bool, str]
 
-As of this PEP - that is, we may expand the flexibility of concatenation in future PEPs - prefixing is the only form of concatenation supported. (That is, the type variable tuple must appear last in the type parameter list.)
-
 ``*args`` as a Type Variable Tuple
 ----------------------------------
 
@@ -348,9 +374,10 @@ instance is *not* allowed:
 
     def foo(*args: Ts): ...  # NOT valid
 
-Also note that if the type variable tuple is wrapped in a ``Tuple``,
-the old behaviour still applies: all arguments must be a ``Tuple``
-parameterised with the same types.
+``*args`` is the only case where an argument can be annotated as ``*Ts`` directly;
+other arguments should use ``*Ts`` to parameterise something else, e.g. ``Tuple[*Ts]``.
+If ``*args`` itself is annotated as ``Tuple[*Ts]``, the old behaviour still applies:
+all arguments must be a ``Tuple`` parameterised with the same types.
 
 ::
 
@@ -401,7 +428,7 @@ Type variable tuples can also be used in the arguments section of a
 
 However, note that as of this PEP, if a type variable tuple does appear in
 the arguments section of a ``Callable``, it must appear alone.
-That is, `Type Prefixing`_ is not supported in the context of ``Callable``.
+That is, `Type Concatenation`_ is not supported in the context of ``Callable``.
 (Use cases where this might otherwise be desirable are likely covered through use
 of either a ``ParamSpec`` from PEP 612, or a type variable tuple in the ``__call__``
 signature of a callback protocol from PEP 544.)
@@ -473,7 +500,7 @@ Normal ``TypeVar`` instances can also be used in such aliases:
     # T is bound to `int`; Ts is bound to `bool, str`
     Foo[int, bool, str]
 
-Note that the same rules for `Type Prefixing`_ apply for aliases.
+Note that the same rules for `Type Concatenation`_ apply for aliases.
 In particular, only one ``TypeVarTuple`` may occur within an alias,
 and the ``TypeVarTuple`` must be at the end of the alias.
 
@@ -552,11 +579,6 @@ in a complete array type.
     # E.g. Array1D[np.uint8]
     Array1D = Array[DataType, Ndim[Literal[1]]]
 
-Final Notes
-===========
-
-**Slice expressions**: type variable tuples may *not* appear in slice expressions.
-
 Rationale and Rejected Ideas
 ============================
 
@@ -594,6 +616,59 @@ otherwise imply. Also, we may later wish to support arguments that should not be
 
 We therefore settled on ``TypeVarTuple``.
 
+Behaviour when Type Parameters are not Specified
+------------------------------------------------
+
+In order to support gradual typing, this PEP states that *both*
+of the following examples should type-check correctly:
+
+::
+
+    def takes_any_array(x: Array): ...
+    x: Array[Height, Width]
+    takes_any_array(x)
+
+    def takes_specific_array(y: Array[Height, Width]): ...
+    y: Array
+    takes_specific_array(y)
+
+Note that this is in contrast to the behaviour of the only existing
+variadic type in Python, ``Tuple``:
+
+::
+
+    def takes_any_tuple(x: Tuple): ...
+    x: Tuple[int, str]
+    takes_any_tuple(x)  # Valid
+
+    def takes_specific_tuple(y: Tuple[int, str]): ...
+    y: Tuple
+    takes_specific_tuple(y)  # Error
+
+The rules for ``Tuple`` were deliberately chosen such that the latter case
+is an error: it was thought to be more likely that the programmer has made a
+mistake than that the function expects a specific kind of ``Tuple`` but the
+specific kind of ``Tuple`` passed is unknown to the type checker. Additionally,
+``Tuple`` is something of a special case, in that it is used to represent
+immutable sequences. That is, if an object's type is inferred to be an
+unparameterised ``Tuple``, it is not necessarily because of incomplete typing.
+
+In contrast, if an object's type is inferred to be an unparameterised ``Array``,
+it is much more likely that the user has simply not yet fully annotated their
+code, or that the signature of a shape-manipulating library function cannot yet
+be expressed using the typing system and therefore returning a plain ``Array``
+is the only option. We rarely deal with arrays of truly arbitrary shape;
+in certain cases, *some* parts of the shape will be arbitrary - for example,
+when dealing with sequences, the first two parts of the shape are often
+'batch' and 'time' - but we plan to support these cases explicitly in a
+future PEP with a syntax such as ``Array[Batch, Time, ...]``.
+
+We therefore made the decision to have variadic generics *other* than
+``Tuple`` behave differently, in order to give the user more flexibility
+in how much of their code they wish to annotate, and to enable compatibility
+between old unannotated code and new versions of libraries which do use
+these type annotations.
+
 Backwards Compatibility
 =======================
 
@@ -606,7 +681,8 @@ TODO
 Reference Implementation
 ========================
 
-TODO
+Two reference implementations exist: one in Pyre, as of TODO, and one in
+Pyright, as of v1.1.108.
 
 Footnotes
 ==========