Huge thank you to everyone who helped get this across the finish line, especially @gvanrossum! Well done to all of us, @mrkmndz @pradeep90 @vsiles @fylux!
A number of cleanups also happened, improving readability of various sections and examples, but the only *semantic* change here is that TypeVarTuples can now be substituted with unbounded tuples (previously that was explicitly forbidden, but we convinced ourselves that there is no reason for that).
These sections were requested in some feedback we got when talking to NumPy and JAX folks. OK, this does make an already-long PEP even longer, but I do think this context is important - especially for folks who don't have strong intuitions for how parametric typing works in Python.
This is the more-or-less final version now. I'll do one last read-through to confirm everything's consistent, then post in typing-sig to confirm it's ready for the Steering Committee.
Semantic changes:
* Remove support for type variable tuples in `Union`. Apparently the implementation would be tricky in Pyre because of special-casing around `Union`. That could be evidence that it would be tricky in Mypy and pytype. Since we don't have a specific use-case in mind, I think this is fine.
* Support concatenation in `Callable`. Pradeep pointed ou that `ParamSpec` doesn't cover all potential use-cases because it doesn't support concatenating suffixes (in turn because `ParamSpec`s can contain keyword arguments, and concatenation is positional).
Readability changes:
* Remove the section on a full `Array` example. We've moved the important thing here - an explicit confirmation that `Array` can be generic in both datatype and shape - to the beginning of the PEP to make it more obvious. We've removed the `Ndim` thing after someone pointed out that e.g. `Ndim[Literal[2]]` would be incompatible with `Shape[Any, Any]`; I just don't think it's going to work. All that's left is the aliases, which we've moved into the Aliases section itself.
Other changes:
* Fill out Backwards Compatibility section.
* Reference initial CPython implementation.
Readability changes:
* Tweak framing to also acknowledge that variadic generics have been wanted for other things too (though as far as I've been able to tell there isn't a consistent enough pattern to the other use cases to justify including them in the PEP)
* Move the Array example much earlier (both because reading a detailed example is going to be a faster way of getting the gist for people in a hurry, and because we want to make it clear to numerical computing folks from the start that we can also specify a datatype here). (We still have a more detailed Array example later; I'm leaving that there for the time being pending some discussion with Pradeep, but eventually I guess it'll be removed and absorbed into this earlier example)
* Make it clear that other types can appear in `Union` alongside `TypeVarTuple` instances.
Semantic changes:
* Disallow a `Union` of more than one type variable tuple (I thought this would be easier than concatenation of more than one type variable tuple, but Pradeep points out it could still get hairy for type checkers, so disallow it for now to be on the safe safe)
Also some *more* type fixes...
Semantic changes:
* Unparameterised variadics are compatible with parameterised variadics in both direction
* Allow suffixing
Readability changes:
* De-emphasise unparameterised `Array` behaving like `Array[Any, ...]` to avoid the suggestion that `Array[Any, ...]` might be supported
* Explicitly forbid `Array[Any, ...]` and similar
* Clarify that only `*args` can be annotated as `*Ts`; other arguments must use `Tuple[*Ts]`
* Remove 'Final Notes' section (I was hoping we'd have more edge cases to add to this, but in the end there haven't been any, and the only example we do have is obscure enough that I think we can just remove it)
Also some minor typo fixes.
* PEP 646: Simplify based on typing-sig discussions
Feature removals:
* Remove `Map` (to be added in a separate PEP)
* Remove support for all forms of concatenation other than simply prefixing of a single `TypeVarTuple` (fancier forms of concatenation to also be added in a separate PEP)
Semantic changes:
* Try switching `Ts` back to mean the 'unpacked' version (such that `Ts = int, str; Tuple[Ts] = Tuple[int, str]`) to see whether everything still works. (Might revert this, pending future discussion in typing-sig.)
* PEP 646: Clarify that prefixing is not allowed in Callable
* Update PEP 646 based on discussions in typing-sig
Semantic changes:
* Switch back to `Ts` meaning types packed in a `Tuple`
* Disallow unpacked usages of `Ts`
* Unparameterised `Tensor` behaves like `Tensor[Any, ...]`
Clarifications:
* `TypeVarTuple`s are invariant
* `TypeVarTuple`s can not be bound to unknown-length types (e.g. `Tuple[int, ...]`
* `TypeVarTuple`s can not be bound to unions of types
* Empty `*args` behaves like `Tuple[()]`
* `TypeVarTuple`s can not appear in slices
Readability changes:
* Reorder some sections for better flow
* Update PEP 646 with Eric's suggestions and other tweaks
Semantic changes:
* Revert to `*args: *Ts` for new behaviour, and `*args: Tuple[*Ts]` for old behaviour
Fixes:
* Unpacking in Aliases/Ideal Array Type sections
* Various typo/grammar fixes
Other:
* State explicitly that TypeVarTuples can be used in callback protocols
Changes detected by Topy (https://github.com/intgr/topy), all changes
verified by hand, false positives have been omitted.
These range from straight-out misspellings to debatable hyphen placement.
The hyphen changes are supported by grammar manuals of style.
Naming:
* Rename Tuple
* Rename Tensor -> Array (it's less jargony, and I think it's the more general term)
* Rename Expand -> Unpack (to be consistent with the terminology for normal tuples)
* Rename ArgTs -> Ts, ReturnT -> R
* Rename 'type tuple variable' -> 'type variable tuple'
Semantic:
* Explicitly state that a TypeVarTuple can hold *zero* or more types
* Explicitly state that TypeVarTuple doesn't support variance or bound yet
* Support Union[*Ts]
* Support concatenating multiple unpacked TypeVarTuple when there's no ambiguity
* Support aliases
* Remove support for class overloads; replace with overloads of individual methods
Pedagogical:
* Reorder introductory material to make it clearer how TypeVarTuples behave when not unpacked, and to make it clearer that using them without unpacking them is a perfectly valid thing to do
* Remove example of unpacking being used with a regular Tuple rather than a TypeVarTuple (my main reason for wanting to include it was to emphasise that a TypeVarTuple behaves like a Tuple, but I think this is emphasised better in previous sections now, and since I don't think it has any use cases, it seems better to remove it to keep things shorter and more to the point)
* Replace Tuple[*Ts] with just Ts (now that we've settled on Ts definitely meaning "A Tuple filled with types", writing Tuple[*Ts] is redundant - it's exactly the same as Ts, but with more keystrokes)
* State explicitly that TypeVarTuple can be used with Callable
* Changes args_to_tuples example to args_to_lists (so it's clearer where the Tuple comes from)
* Show more examples of where Map can be used
* Move section on nesting Map to the 'Rationale and Rejected Ideas' section (it's complicated enough to be too distracting if it were in the main section, and since there isn't an obvious use-case, we leave it as an optional feature)
* Add section on a full example of an array type
* Remove ideas for future PEPs (to reduce length)
* Add more detail on the range of type concatenations that are allowable
Other:
* Add Pradeep to the authors list, since he's contributed so much :)
* Add Eric Traut in the acknowledgements
* Update Post-History
* Fix some references
* Fix first Array example to remove the need for a cast
* Various wording tweaks
(This draft will need some work before it is acceptable, see my message to typing-sig and my comments in the Google Doc. But I'd like to claim the PEP number.)
Petr Viktorin
Adam Turner
Matthew Rahtz
Pradeep Kumar
Jelle Zijlstra
Dimitri Papadopoulos Orfanos
Eric Traut
Hugo van Kemenade
Marti Raudsepp
Ken Jin