PEP 646: Add section on grammar changes (#2039)

pep-0646.rst

@@ -733,29 +733,209 @@ hopefully enable a thriving ecosystem of tools for analysing and verifying
 shape properties of numerical computing programs.
 
 
+Grammar Changes
+===============
+
+This PEP requires two grammar changes. Full diffs of ``python.gram``
+and simple tests to confirm correct behaviour are available at
+https://github.com/mrahtz/cpython/commits/pep646-grammar.
+
+Star expressions in indexes
+---------------------------
+
+The first grammar change enables use of star expressions in index operations (that is,
+within square brackets), necessary to support star-unpacking of TypeVarTuples:
+
+::
+
+    DType = TypeVar('DType')
+    Shape = TypeVarTuple('Shape')
+    class Array(Generic[DType, *Shape]):
+        ...
+
+Before:
+
+::
+    
+    slices:
+        | slice !','
+        | ','.slice+ [',']
+
+After:
+
+::
+    
+    slices:
+        | slice !','
+        | ','.(slice | starred_expression)+ [',']
+
+As with star-unpacking in other contexts, the star operator calls ``__iter__``
+on the callee, and adds the contents of the resulting iterator to the argument
+passed to ``__getitem__``. For example, if we do ``foo[a, *b, c]``, and
+``b.__iter__`` produces an iterator yielding ``d`` and ``e``,
+``foo.__getitem__`` would receive ``(a, d, e, c)``.
+
+To put it another way, note that ``x[..., *a, ...]`` produces the same result
+as ``x[(..., a*, ...)]``` (with any slices ``i:j`` in ``...`` replaced with
+``slice(i, j)``, with the one edge case that ``x[*a]`` becomes ``x[(*a,)]``).
+
+TypeVarTuple Implementation
+'''''''''''''''''''''''''''
+
+With this grammar change, ``TypeVarTuple`` is implemented as follows.
+Note that this implementation is useful only for the benefit of a) correct
+``repr()`` and b) runtime analysers; static analysers would not use the
+implementation.
+
+::
+
+    class TypeVarTuple:
+        def __init__(self, name):
+            self._name = name
+            self._unpacked = UnpackedTypeVarTuple(name) 
+        def __iter__(self):
+            yield self._unpacked
+        def __repr__(self):
+            return self._name
+
+    class UnpackedTypeVarTuple:
+        def __init__(self, name):
+            self._name = name
+        def __repr__(self):
+            return '*' + self._name
+
+Implications
+''''''''''''
+
+This grammar change implies a number of additional changes in behaviour not
+required by this PEP. We choose to allow these additional changes rather than
+disallowing them at a syntax level in order to keep the syntax change as small
+as possible.
+
+First, the grammar change enables star-unpacking of other structures, such
+as lists, within indexing operations:
+
+::
+
+    idxs_to_select = (1, 2)
+    array[0, *idxs_to_select, -1]  # Equivalent to [0, 1, 2, -1]
+
+Second, more than one instance of a star-unpack can occur within an index:
+
+::
+
+    array[*idxs_to_select, *idxs_to_select]  # Equivalent to array[1, 2, 1, 2]
+
+Note that this PEP disallows multiple unpacked TypeVarTuples within a single
+type parameter list. This requirement would therefore need to be implemented
+in type checking tools themselves rather than at the syntax level.
+
+Third, slices may co-occur with starred expressions:
+
+::
+
+    array[3:5, *idxs_to_select]  # Equivalent to array[3:5, 1, 2]
+
+However, note that slices involving starred expressions are still invalid:
+
+::
+
+    # Syntax error
+    array[*idxs_start:*idxs_end]
+
+
+``*args`` as a TypeVarTuple
+---------------------------
+
+The second change enables use of ``*args: *Ts`` in function definitions.
+
+Before:
+
+::
+
+    star_etc:
+    | '*' param_no_default param_maybe_default* [kwds]
+    | '*' ',' param_maybe_default+ [kwds]
+    | kwds
+
+After:
+
+::
+
+    star_etc:
+    | '*' param_no_default param_maybe_default* [kwds]
+    | '*' param_no_default_star_annotation param_maybe_default* [kwds]  # New
+    | '*' ',' param_maybe_default+ [kwds]
+    | kwds
+
+Where:
+
+::
+
+    param_no_default_star_annotation:
+    | param_star_annotation ',' TYPE_COMMENT?
+    | param_star_annotation TYPE_COMMENT? &')'
+
+    param_star_annotation: NAME star_annotation
+
+    star_annotation: ':' star_expression
+
+This accomplishes the desired outcome (making ``*args: *Ts`` not be a syntax
+error) while matching the behaviour of star-unpacking in other contexts:
+at runtime, ``__iter__`` is called on the starred object, and a tuple
+containing the items of the resulting iterator is set as the type annotion
+for ``args``. In other words, at runtime ``*args: *foo`` is equivalent to
+``*args: tuple(foo)``.
+
+::
+
+    >>> Ts = TypeVarTuple('Ts')
+    >>> def foo(*args: *Ts): pass  # Equivalent to `*args: tuple(Ts)`
+    >>> foo.__annotations__
+    {'args': (*Ts,)}
+    # *Ts is the repr() of Ts._unpacked, an instance of UnpackedTypeVarTuple
+
+Note that the only scenario in which this grammar change allows ``*Ts`` to be
+used as a direct annotation (rather than being wrapped in e.g. ``Tuple[*Ts]``)
+is ``*args``. Other uses are still invalid:
+
+::
+
+    x: *Ts                 # Syntax error
+    def foo(x: *Ts): pass  # Syntax error
+
+Implications
+''''''''''''
+
+As with the first grammar change, this change also has a number of side effects.
+In particular, the annotation of ``*args`` could be set to a starred object
+other than a ``TypeVarTuple`` - for example, the following nonsensical
+annotation is possible:
+
+::
+
+    >>> foo = [1, 2, 3]
+    >>> def bar(*args: *foo): pass  # Equivalent to `*args: tuple(foo)`
+    >>> bar.__annotations__
+    {'args': (1, 2, 3)}
+
+Again, prevention of such annotations will need to be done by, say, static
+checkers, rather than at the level of syntax.
+
+Alternatives
+------------
+
+If these two grammar changes are considered too burdensome, there are two alternatives:
+
+1. **Support change 1 but not change 2**. Starred expressions within indexes are
+   more important to us than the ability to annotation ``*args``.
+2. **Use ``Unpack`` instead** - though in this case it might be better for us to
+   reconsider our options to see whether there isn't another option which would be
+   more readable.
+
 Backwards Compatibility
 =======================
 
-In order to use the star operator for unpacking of ``TypeVarTuple`` instances,
-we would need to make two grammar changes:
-
-1. Star expressions must be made valid in at least index operations.
-   For example, ``Tuple[*Ts]`` and ``Tuple[T1, *Ts, T2]`` would both
-   be valid. (This PEP does not allow multiple unpacked ``TypeVarTuple``
-   instances to appear in a single parameter list, so ``Tuple[*Ts1, *Ts2]``
-   would be a runtime error. Also note that star expressions would *not*
-   be valid in slice expressions -  e.g. ``Tuple[*Ts:*Ts]`` is
-   nonsensical and should remain invalid.)
-2. We would need to make '``*args: *Ts``' valid in function definitions.
-
-In both cases, at runtime the star operator would call ``Ts.__iter__()``.
-This would, in turn, return an instance of a helper class, e.g.
-``UnpackedTypeVarTuple``, whose ``repr`` would be ``*Ts``.
-
-If these grammar changes are considered too burdensome, we could instead
-simply use ``Unpack`` - though in this case it might be better for us to
-first decide whether there's a better option.
-
 The ``Unpack`` version of the PEP should be back-portable to previous
 versions of Python.
 
@@ -766,7 +946,6 @@ uses of the class will still work, and b) parameterised and unparameterised
 versions of the class can be used together (relevant if, for example, library
 code is updated to use parameters while user code is not, or vice-versa).
 
-
 Reference Implementation
 ========================