PEP 695: Incorporate feedback (#2706)

Incorporated changes from latest round of feedback, and describe new implementation strategy.

Co-authored-by: Eric Traut <erictr@microsoft.com>

pep-0695.rst

@@ -91,10 +91,10 @@ equivalent.
 Type variables defined within the global scope also need to be given a name
 that starts with an underscore to indicate that the variable is private to
 the module. Globally-defined type variables are also often given names to
-indicate their variance leading to cumbersome names like "_T_contra" and
+indicate their variance, leading to cumbersome names like "_T_contra" and
 "_KT_co". The current mechanisms for allocating type variables also requires
 the developer to supply a redundant name in quotes (e.g. ``T = TypeVar("T")``).
-This PEP eliminates the need for the redundant name and the cumbersome
+This PEP eliminates the need for the redundant name and cumbersome
 variable names.
 
 Defining type parameters today requires importing the ``TypeVar`` and 
@@ -225,6 +225,20 @@ time.
     def func2[T](T): ... # Syntax Error
 
 
+Class type parameter names are not mangled if they begin with a double
+underscore. Mangling would not make sense because type parameters, unlike other
+class-scoped variables, cannot be accessed through the class dictionary, and
+the notion of a "private" type parameter doesn't make sense. Other class-scoped
+variables are mangled if they begin with a double underscore, so the mangled
+name is used to determine whether there is a name collision with type parameters.
+
+::
+
+    class ClassA[__T, _ClassA__S]:
+        __T = 0  # OK
+        __S = 0  # Syntax Error (because mangled name is _ClassA__S)
+
+
 Type Parameter Scopes
 ---------------------
 
@@ -393,7 +407,8 @@ At runtime, a ``type`` statement will generate an instance of
 include:
 
 * ``__name__`` is a str representing the name of the type alias
-* ``__parameters__`` is a tuple of ``TypeVar``, ``TypeVarTuple``, or ``ParamSpec`` objects
+* ``__parameters__`` is a tuple of ``TypeVar``, ``TypeVarTuple``, or
+    ``ParamSpec`` objects that parameterize the type alias if it is generic
 * ``__value__`` is the evaluated value of the type alias
 
 The ``__value__`` attribute initially has a value of ``None`` while the type
@@ -606,47 +621,47 @@ that includes a list of type parameters associated with the function or class.
 Compiler Changes
 ----------------
 
-The compiler maintains a list of "active type parameters" as it recursively
+The compiler maintains a list of "active type variables" as it recursively
 generates byte codes for the program. Consider the following example.
 
 ::
 
     class Outer[K, V]:
-        # Active type parameters are K and V
+        # Active type variables are K and V
 
         class Inner[T]:
-            # Active type parameters are K, V, and T
+            # Active type variables are K, V, and T
 
             def method[M](self, a: M) -> M:
-                # Active type parameters are K, V, T, and M
+                # Active type variables are K, V, T, and M
                 ...
 
-An active type parameter symbol cannot be used for other purposes within
+An active type variable symbol cannot be used for other purposes within
 these scopes. This includes local parameters, local variables, variables
 bound from other scopes (nonlocal or global), or other type parameters. An
-attempt to reuse a type parameter name in one of these manners results in
+attempt to reuse a type variable name in one of these manners results in
 a syntax error.
 
 ::
 
     class ClassA[K, V]:
-        class Inner[K]: # Syntax error: K already in use as type parameter
+        class Inner[K]: # Syntax error: K already in use as type variable
             ...
 
     class ClassB[K, V]:
-        def method(self, K): # Syntax error: K already in use as type parameter
+        def method(self, K): # Syntax error: K already in use as type variable
             ...
 
-    class ClassC[T, T]: # Syntax error: T already in use as type parameter
+    class ClassC[T, T]: # Syntax error: T already in use as type variable
         ...
 
     def func1[T]():
         ...
 
 
-A type parameter is considered "active" when compiling the arguments for
+A type variable is considered "active" when compiling the arguments for
 a class declaration, the type annotations for a function declaration, and
-the right-hand expression in a type alias declaration. Type parameters are
+the right-hand expression in a type alias declaration. Type variable are
 not considered "active" when compiling the default argument expressions for
 a function declaration or decorator expressions for classes or functions.
 
@@ -655,39 +670,67 @@ a function declaration or decorator expressions for classes or functions.
     T = list
 
     @decorator(T) # T in decorator refers to outer variable
-    class ClassA[T](Base[T], metaclass=Meta[T]) # T refers to type parameter
+    class ClassA[T](Base[T], metaclass=Meta[T]) # T refers to type variable
         ...
 
     @decorator(T) # T in decorator refers to outer variable
-    def func1[T](a: list[T]) -> T: # T refers to type parameter
+    def func1[T](a: list[T]) -> T: # T refers to type variable
         ...
 
     def func2[T](a = T): # T in default refers to outer variable
         ...
 
 
-When a type parameter is referenced, the compiler generates the byte codes
+When a ``class``, ``def``, or ``type`` statement includes one or more type
-to construct a ``typing.TypeParameter`` object. This new class can be thought
+parameters, the compiler emits byte codes to construct the corresponding
-of as a proxy for ``TypeVar``, ``TypeVarTuple`` or ``ParamSpec``. It has a
+``typing.TypeVar``, ``typing.TypeVarTuple``, or ``typing.ParamSpec`` instances.
-name and ``args`` and ``kwargs`` properties (required for ``ParamSpec``) so
+It then builds a new tuple that includes all active type variables and stores
-it can be used within a type expression at runtime. It also implements a
+this new tuple in a local variable. Active type variables include all type
-``__or__`` and ``__ror__`` method so it can be used in unions, an ``__iter__``
+parameters declared by outer ``class`` and ``def`` scopes plus those declared
-method so it can be unpacked (needed for ``TypeVarTuple``) and a ``__repr__``
+by the ``class``, ``def``, or ``type`` statement itself. (In the reference
-method so its name can be printed. It does not allow for runtime differentiation
+implementation, the local variable happens to have the name
-between a ``TypeVar``, ``TypeVarTuple`` or ``ParamSpec``, nor does it
+``__type_variables__``, but this is an implementation detail. Other Python
-allow for runtime introspection of the upper bound, constraints, or
+compilers or future versions of the CPython compiler may choose a different
-variance.
+name or an entirely anonymous local variable slot for this purpose.)
 
-If a bound expression is provided for a type parameter, it is validated for
+When a type variable is referenced, the compiler generates opcodes that
-syntax but is not evaluated at runtime. That is, no byte codes are generated
+load the active type variable tuple from either the local variable or (if
-for the bound expression. Type checkers should evaluate its type statically.
+there are no local type variables) through the use of a new opcode called
+``LOAD_TYPEVARS`` that loads the tuple of active type variables from the
+current function object. It then emits opcodes to index into this tuple to
+fetch the desired type variable.
+
+When a new function is created, the "active type variables" tuple is copied
+to the C struct field ``func_typevars`` of the function object, making the type
+variables from outer scopes available to inner scopes of the function or class.
+
+A new read-only attribute called ``__type_variables__`` is available on class,
+function, and type alias objects. This attribute is a tuple of the active
+type variables that are visible within the scope of that class, function,
+or type alias. This attribute is used for runtime evaluation of stringified
+(forward referenced) type annotations that include references to type
+parameters. Functions like ``typing.get_type_hints`` can use this attribute
+to populate the ``locals`` dictionary with values for type parameters that
+are in scope when calling ``eval`` to evaluate the stringified expression.
+
+
+Library Changes
+---------------
+
+Several classes in the ``typing`` module that are currently implemented in
+Python must be reimplemented in C. This includes: ``TypeVar``,
+``TypeVarTuple``, ``ParamSpec``, and ``Generic``. The new class
+``TypeAliasType`` (described above) also must be implemented in C.
+
+The ``typing.get_type_hints`` must be updated to use the new
+``__type_variables__`` attribute.
 
 
 Reference Implementation
 ========================
 
 This proposal is prototyped in the CPython code base in
-`this fork <https://github.com/erictraut/cpython/tree/type_param_syntax>`_.
+`this fork <https://github.com/erictraut/cpython/tree/type_param_syntax2>`_.
 
 The Pyright type checker supports the behavior described in this PEP.
 
@@ -712,12 +755,16 @@ function decorators, which are common in Python. Only one other popular
 programming language, C++, uses this approach.
 
 
-Extends Keyword
+Bounds Syntax
----------------
+-------------
-We considered introducing an "extends" keyword for specifying the upper bound
+We explored various syntactic options for specifying the bounds and constraints
-of a type parameter. This is consistent with Java and TypeScript, two other
+for a type variable. We considered, but ultimately rejected, the use
-popular languages. However, "extends" didn't feel very Pythonic. It also didn't
+of a ``<:`` token like in Scala, the use of an ``extends`` or ``with``
-lend itself to supporting constrained type parameters.
+keyword like in various other languages, and the use of a function call
+syntax similar to today's ``typing.TypeVar`` constructor. The simple colon
+syntax is consistent with many other programming languages (see Appendix A),
+and it was heavily preferred by a cross section of Python developers who
+were surveyed.
 
 
 Explicit Variance
@@ -740,6 +787,19 @@ compiler. This mangled name would be based on the qualified name of the
 generic class, function or type alias it was associated with. This approach
 was rejected because qualified names are not necessarily unique, which means
 the mangled name would need to be based on some other randomized value.
+Furthermore, this approach is not compatible with techniques used for
+evaluating quoted (forward referenced) type annotations.
+
+
+Lambda Lifting
+--------------
+When considering implementation options, we considered introducing a new
+scope and executing the ``class``, ``def``, or ``type`` statement within
+a lambda -- a technique that is sometimes referred to as "lambda lifting".
+We ultimately rejected this idea because it did not work well for statements
+within a class body (because class-scoped symbols cannot be accessed by
+inner scopes). It also introduced many odd behaviors for scopes that were
+further nested within the lambda.
 
 
 Appendix A: Survey of Type Parameter Syntax
@@ -748,7 +808,10 @@ Appendix A: Survey of Type Parameter Syntax
 Support for generic types is found in many programming languages. In this
 section, we provide a survey of the options used by other popular programming
 languages. This is relevant because familiarity with other languages will
-make it easier for Python developers to understand this concept.
+make it easier for Python developers to understand this concept. We provide
+additional details here (for example, default type argument support) that
+may be useful when considering future extensions to the Python type system.
+
 
 C++
 ---
@@ -1098,6 +1161,16 @@ Summary
 +------------+----------+---------+--------+----------+-----------+-----------+
 
 
+Acknowledgements
+================
+
+Thanks to Sebastian Rittau for kick-starting the discussions that led to this
+proposal, to Jukka Lehtosalo for proposing the syntax for type alias
+statements and to Jelle Zijlstra, Daniel Moisset, and Guido van Rossum
+for their valuable feedback and suggested improvements to the specification
+and implementation.
+
+
 Copyright
 =========