[pep 563] Second public draft to be discussed on python-dev.

pep-0563.rst

@@ -70,10 +70,13 @@ checking, it is worth mentioning that the design of this PEP, as well
 as its precursors (PEP 484 and PEP 526), is predominantly motivated by
 the type hinting use case.
 
-With Python 3.7, PEP 484 graduates from provisional status.  Other
+In Python 3.8 PEP 484 will graduate from provisional status.  Other
 enhancements to the Python programming language like PEP 544, PEP 557,
 or PEP 560, are already being built on this basis as they depend on
 type annotations and the ``typing`` module as defined by PEP 484.
+In fact, the reason PEP 484 is staying provisional in Python 3.7 is to
+enable rapid evolution for another release cycle that some of the
+aforementioned enhancements require.
 
 With this in mind, uses for annotations incompatible with the
 aforementioned PEPs should be considered deprecated.
@@ -88,10 +91,10 @@ in the respective ``__annotations__`` dictionary.  Static type checkers
 will see no difference in behavior, whereas tools using annotations at
 runtime will have to perform postponed evaluation.
 
-If an annotation was already a string, this string is preserved
+The string form is obtained from the AST during the compilation step,
-verbatim.  In other cases, the string form is obtained from the AST
+which means that the string form might not preserve the exact formatting
-during the compilation step, which means that the string form preserved
+of the source.  Note: if an annotation was a string literal already, it
-might not preserve the exact formatting of the source.
+will still be wrapped in a string.
 
 Annotations need to be syntactically valid Python expressions, also when
 passed as literal strings (i.e. ``compile(literal, '', 'eval')``).
@@ -110,6 +113,9 @@ The functionality described above can be enabled starting from Python
 
     from __future__ import annotations
 
+A reference implementation of this functionality is available
+`on GitHub <https://github.com/ambv/cpython/tree/string_annotations>`_.
+
 
 Resolving Type Hints at Runtime
 ===============================
@@ -129,10 +135,10 @@ annotation.
 
 In both cases it's important to consider how globals and locals affect
 the postponed evaluation.  An annotation is no longer evaluated at the
-time of definition and, more importantly, *in the same scope* it was
+time of definition and, more importantly, *in the same scope* where it
-defined.  Consequently, using local state in annotations is no longer
+was defined.  Consequently, using local state in annotations is no
-possible in general.  As for globals, the module where the annotation
+longer possible in general.  As for globals, the module where the
-was defined is the correct context for postponed evaluation.
+annotation was defined is the correct context for postponed evaluation.
 
 The ``get_type_hints()`` function automatically resolves the correct
 value of ``globalns`` for functions and classes.  It also automatically
@@ -236,17 +242,33 @@ valid.  They can use local names or the fully qualified name.  Example::
 
     class C:
         field = 'c_field'
-        def method(self, arg: C.field) -> None:  # this is OK
+        def method(self) -> C.field:  # this is OK
+            ...
+
+        def method(self) -> field:  # this is OK
+            ...
+
+        def method(self) -> C.D:  # this is OK
+            ...
+
+        def method(self) -> D:  # this is OK
             ...
 
         class D:
             field2 = 'd_field'
-            def method(self, arg: C.field) -> C.D.field2:  # this is OK
+            def method(self) -> C.D.field2:  # this is OK
                 ...
 
-            def method(self, arg: field) -> D.field2:  # this is OK
+            def method(self) -> D.field2:  # this is OK
                 ...
 
+            def method(self) -> field2:  # this is OK
+                ...
+
+            def method(self) -> field:  # this FAILS, class D doesn't
+                ...                     # see C's attributes,  This was
+                                        # already true before this PEP.
+
 In the presence of an annotation that isn't a syntactically valid
 expression, SyntaxError is raised at compile time.  However, since names
 aren't resolved at that time, no attempt is made to validate whether
@@ -284,22 +306,23 @@ forward references with string literals.  The list includes:
 
 * type definitions::
 
-    T = TypeVar('T', bound='UserId')
+    T = TypeVar('T', bound='<type>')
-    UserId = NewType('UserId', 'SomeType')
+    UserId = NewType('UserId', '<type>')
-    Employee = NamedTuple('Employee', [('name', str), ('id', 'UserId')])
+    Employee = NamedTuple('Employee', [('name', '<type>', ('id', '<type>')])
 
 * aliases::
 
-    Alias = Optional['SomeType']
+    Alias = Optional['<type>']
-    AnotherAlias = Union['SomeType', 'OtherType']
+    AnotherAlias = Union['<type>', '<type>']
+    YetAnotherAlias = '<type>'
 
 * casting::
 
-    cast('SomeType', value)
+    cast('<type>', value)
 
 * base classes::
 
-    class C(Tuple['SomeType', 'OtherType']): ...
+    class C(Tuple['<type>', '<type>']): ...
 
 Depending on the specific case, some of the cases listed above might be
 worked around by placing the usage in a ``if TYPE_CHECKING:`` block.
@@ -316,8 +339,8 @@ language and is out of scope for this PEP.
 Rejected Ideas
 ==============
 
-Keep the ability to use function local state when defining annotations
+Keeping the ability to use function local state when defining annotations
-----------------------------------------------------------------------
+-------------------------------------------------------------------------
 
 With postponed evaluation, this would require keeping a reference to
 the frame in which an annotation got created.  This could be achieved
@@ -327,6 +350,11 @@ This would be prohibitively expensive for highly annotated code as the
 frames would keep all their objects alive. That includes predominantly
 objects that won't ever be accessed again.
 
+To be able to address class-level scope, the lambda approach would
+require a new kind of cell in the interpreter.  This would proliferate
+the number of types that can appear in ``__annotations__``, as well as
+wouldn't be as introspectable as strings.
+
 Note that in the case of nested classes, the functionality to get the
 effective "globals" and "locals" at definition time is provided by
 ``typing.get_type_hints()``.
@@ -336,15 +364,15 @@ have to use local variables, it can populate the given generated
 object's ``__annotations__`` dictionary directly, without relying on
 the compiler.
 
-Disallow local state usage for classes, too
+Disallowing local state usage for classes, too
--------------------------------------------
+----------------------------------------------
 
 This PEP originally proposed limiting names within annotations to only
 allow names from the model-level scope, including for classes.  The
 author argued this makes name resolution unambiguous, including in cases
 of conflicts between local names and module-level names.
 
-This idea was ultimately rejected in case of nested classes.  Instead,
+This idea was ultimately rejected in case of classes.  Instead,
 ``typing.get_type_hints()`` got modified to populate the local namespace
 correctly if class-level annotations are needed.
 
@@ -355,8 +383,8 @@ local scope access is required for class decorators to be able to
 evaluate type annotations. This is because class decorators are applied
 before the class receives its name in the outer scope.
 
-Introduce a new dictionary for the string literal form instead
+Introducing a new dictionary for the string literal form instead
---------------------------------------------------------------
+----------------------------------------------------------------
 
 Yury Selivanov shared the following idea:
 
@@ -384,8 +412,8 @@ recognize that forward references are allowed.
 Finally, just-in-time evaluation in ``__annotations__`` is an
 unnecessary step if ``get_type_hints()`` is used later.
 
-Drop annotations with -O
+Dropping annotations with -O
-------------------------
+----------------------------
 
 There are two reasons this is not satisfying for the purpose of this
 PEP.
@@ -404,6 +432,42 @@ a possibility in the future, as it's conceptually compatible with
 existing -O behavior (dropping docstrings and assert statements).  This
 PEP does not invalidate the idea.
 
+Pass string literals in annotations verbatim to ``__annotations__``
+-------------------------------------------------------------------
+
+This PEP originally suggested directly storing the contents of a string
+literal under its respective key in ``__annotations__``.  This was
+meant to simplify support for runtime type checkers.
+
+Mark Shannon pointed out this idea was flawed since it wasn't handling
+situations where strings are only part of a type annotation.
+
+The inconsistency of it was always apparent but given that it doesn't
+fully prevent cases of double-wrapping strings anyway, it is not worth
+it.
+
+Make the name of the future import more verbose
+-----------------------------------------------
+
+Instead of requiring the following import::
+
+    from __future__ import annotations
+
+the PEP could call the feature more explicitly, for example
+``string_annotations``, ``stringify_annotations``,
+``annotation_strings``, ``annotations_as_strings``, ``lazy_anotations``,
+``static_annotations``, etc.
+
+The problem with those names is that they are very verbose.  Each of
+them besides ``lazy_annotations`` would constitute the longest future
+feature name in Python.  They are long to type and harder to remember
+than the single-word form.
+
+There is precedence of a future import name that sounds overly generic
+but in practice was obvious to users as to what it does::
+
+    from __future__ import division
+
 
 Prior discussion
 ================
@@ -538,6 +602,62 @@ helper, which could solve self-referencing classes even in the form of a
 class decorator.  He suggested the PEP should provide this helper
 function in the standard library.
 
+Second draft discussion on python-dev
+-------------------------------------
+
+Discussion happened mainly in the `announcement thread <https://mail.python.org/pipermail/python-dev/2017-November/150062.html>`_,
+followed by a brief discussion under `Mark Shannon's post
+<https://mail.python.org/pipermail/python-dev/2017-November/150637.html>`_.
+
+Steven D'Aprano was concerned whether it's acceptable for typos to be
+allowed in annotations after the change proposed by the PEP.  Brett
+Cannon responded that type checkers and other static analyzers (like
+linters or programming text editors) will catch this type of error.
+Jukka Lehtosalo added that this situation is analogous to how names in
+function bodies are not resolved until the function is called.
+
+A major topic of discussion was Nick Coghlan's suggestion to store
+annotations in "thunk form", in other words as a specialized lambda
+which would be able to access class-level scope (and allow for scope
+customization at call time).  He presented a possible design for it
+(`indirect attribute cells
+<https://mail.python.org/pipermail/python-dev/2017-November/150141.html>`_).
+This was later seen as equivalent to "special forms" in Lisp.  Guido van
+Rossum expressed worry that this sort of feature cannot be safely
+implemented in twelve weeks (i.e. in time before the Python 3.7 beta
+freeze).
+
+After a while it became clear that the point of division between
+supporters of the string form vs. supporters of the thunk form is
+actually about whether annotations should be perceived as a general
+syntactic element vs. something tied to the type checking use case.
+
+Finally, Guido van Rossum declared he's rejecting the thunk idea
+based on the fact that it would require a new building block in the
+interpreter.  This block would be exposed in annotations, multiplying
+possible types of values stored in ``__annotations__`` (arbitrary
+objects, strings, and now thunks).  Moreover, thunks aren't as
+introspectable as strings.  Most importantly, Guido van Rossum
+explicitly stated interest in gradually restricting the use of
+annotations to static typing (with an optional runtime component).
+
+Nick Coghlan got convinced to PEP 563, too, promptly beginning
+the mandatory bike shedding session on the name of the ``__future__``
+import.  Many debaters agreed that ``annotations`` seems like
+an overly broad name for the feature name.  Guido van Rossum briefly
+decided to call it ``string_annotations`` but then changed his mind,
+arguing that ``division`` is a precedent of a broad name with a clear
+meaning.
+
+The final improvement to the PEP suggested in the discussion by Mark
+Shannon was the rejection of the temptation to pass string literals
+through to ``__annotations__`` verbatim.
+
+A side-thread of discussion started around the runtime penalty of
+static typing, with topic like the import time of the ``typing``
+module (which is comparable to ``re`` without dependencies, and
+three times as heavy as ``re`` when counting dependencies).
+
 
 Acknowledgements
 ================