diff --git a/pep-0563.rst b/pep-0563.rst index aee113d7b..5177a1ef1 100644 --- a/pep-0563.rst +++ b/pep-0563.rst @@ -70,10 +70,13 @@ checking, it is worth mentioning that the design of this PEP, as well as its precursors (PEP 484 and PEP 526), is predominantly motivated by the type hinting use case. -With Python 3.7, PEP 484 graduates from provisional status. Other +In Python 3.8 PEP 484 will graduate from provisional status. Other enhancements to the Python programming language like PEP 544, PEP 557, or PEP 560, are already being built on this basis as they depend on type annotations and the ``typing`` module as defined by PEP 484. +In fact, the reason PEP 484 is staying provisional in Python 3.7 is to +enable rapid evolution for another release cycle that some of the +aforementioned enhancements require. With this in mind, uses for annotations incompatible with the aforementioned PEPs should be considered deprecated. @@ -88,10 +91,10 @@ in the respective ``__annotations__`` dictionary. Static type checkers will see no difference in behavior, whereas tools using annotations at runtime will have to perform postponed evaluation. -If an annotation was already a string, this string is preserved -verbatim. In other cases, the string form is obtained from the AST -during the compilation step, which means that the string form preserved -might not preserve the exact formatting of the source. +The string form is obtained from the AST during the compilation step, +which means that the string form might not preserve the exact formatting +of the source. Note: if an annotation was a string literal already, it +will still be wrapped in a string. Annotations need to be syntactically valid Python expressions, also when passed as literal strings (i.e. ``compile(literal, '', 'eval')``). @@ -110,6 +113,9 @@ The functionality described above can be enabled starting from Python from __future__ import annotations +A reference implementation of this functionality is available +`on GitHub `_. + Resolving Type Hints at Runtime =============================== @@ -129,10 +135,10 @@ annotation. In both cases it's important to consider how globals and locals affect the postponed evaluation. An annotation is no longer evaluated at the -time of definition and, more importantly, *in the same scope* it was -defined. Consequently, using local state in annotations is no longer -possible in general. As for globals, the module where the annotation -was defined is the correct context for postponed evaluation. +time of definition and, more importantly, *in the same scope* where it +was defined. Consequently, using local state in annotations is no +longer possible in general. As for globals, the module where the +annotation was defined is the correct context for postponed evaluation. The ``get_type_hints()`` function automatically resolves the correct value of ``globalns`` for functions and classes. It also automatically @@ -236,17 +242,33 @@ valid. They can use local names or the fully qualified name. Example:: class C: field = 'c_field' - def method(self, arg: C.field) -> None: # this is OK + def method(self) -> C.field: # this is OK + ... + + def method(self) -> field: # this is OK + ... + + def method(self) -> C.D: # this is OK + ... + + def method(self) -> D: # this is OK ... class D: field2 = 'd_field' - def method(self, arg: C.field) -> C.D.field2: # this is OK + def method(self) -> C.D.field2: # this is OK ... - def method(self, arg: field) -> D.field2: # this is OK + def method(self) -> D.field2: # this is OK ... + def method(self) -> field2: # this is OK + ... + + def method(self) -> field: # this FAILS, class D doesn't + ... # see C's attributes, This was + # already true before this PEP. + In the presence of an annotation that isn't a syntactically valid expression, SyntaxError is raised at compile time. However, since names aren't resolved at that time, no attempt is made to validate whether @@ -284,22 +306,23 @@ forward references with string literals. The list includes: * type definitions:: - T = TypeVar('T', bound='UserId') - UserId = NewType('UserId', 'SomeType') - Employee = NamedTuple('Employee', [('name', str), ('id', 'UserId')]) + T = TypeVar('T', bound='') + UserId = NewType('UserId', '') + Employee = NamedTuple('Employee', [('name', '', ('id', '')]) * aliases:: - Alias = Optional['SomeType'] - AnotherAlias = Union['SomeType', 'OtherType'] + Alias = Optional[''] + AnotherAlias = Union['', ''] + YetAnotherAlias = '' * casting:: - cast('SomeType', value) + cast('', value) * base classes:: - class C(Tuple['SomeType', 'OtherType']): ... + class C(Tuple['', '']): ... Depending on the specific case, some of the cases listed above might be worked around by placing the usage in a ``if TYPE_CHECKING:`` block. @@ -316,8 +339,8 @@ language and is out of scope for this PEP. Rejected Ideas ============== -Keep the ability to use function local state when defining annotations ----------------------------------------------------------------------- +Keeping the ability to use function local state when defining annotations +------------------------------------------------------------------------- With postponed evaluation, this would require keeping a reference to the frame in which an annotation got created. This could be achieved @@ -327,6 +350,11 @@ This would be prohibitively expensive for highly annotated code as the frames would keep all their objects alive. That includes predominantly objects that won't ever be accessed again. +To be able to address class-level scope, the lambda approach would +require a new kind of cell in the interpreter. This would proliferate +the number of types that can appear in ``__annotations__``, as well as +wouldn't be as introspectable as strings. + Note that in the case of nested classes, the functionality to get the effective "globals" and "locals" at definition time is provided by ``typing.get_type_hints()``. @@ -336,15 +364,15 @@ have to use local variables, it can populate the given generated object's ``__annotations__`` dictionary directly, without relying on the compiler. -Disallow local state usage for classes, too -------------------------------------------- +Disallowing local state usage for classes, too +---------------------------------------------- This PEP originally proposed limiting names within annotations to only allow names from the model-level scope, including for classes. The author argued this makes name resolution unambiguous, including in cases of conflicts between local names and module-level names. -This idea was ultimately rejected in case of nested classes. Instead, +This idea was ultimately rejected in case of classes. Instead, ``typing.get_type_hints()`` got modified to populate the local namespace correctly if class-level annotations are needed. @@ -355,8 +383,8 @@ local scope access is required for class decorators to be able to evaluate type annotations. This is because class decorators are applied before the class receives its name in the outer scope. -Introduce a new dictionary for the string literal form instead --------------------------------------------------------------- +Introducing a new dictionary for the string literal form instead +---------------------------------------------------------------- Yury Selivanov shared the following idea: @@ -384,8 +412,8 @@ recognize that forward references are allowed. Finally, just-in-time evaluation in ``__annotations__`` is an unnecessary step if ``get_type_hints()`` is used later. -Drop annotations with -O ------------------------- +Dropping annotations with -O +---------------------------- There are two reasons this is not satisfying for the purpose of this PEP. @@ -404,6 +432,42 @@ a possibility in the future, as it's conceptually compatible with existing -O behavior (dropping docstrings and assert statements). This PEP does not invalidate the idea. +Pass string literals in annotations verbatim to ``__annotations__`` +------------------------------------------------------------------- + +This PEP originally suggested directly storing the contents of a string +literal under its respective key in ``__annotations__``. This was +meant to simplify support for runtime type checkers. + +Mark Shannon pointed out this idea was flawed since it wasn't handling +situations where strings are only part of a type annotation. + +The inconsistency of it was always apparent but given that it doesn't +fully prevent cases of double-wrapping strings anyway, it is not worth +it. + +Make the name of the future import more verbose +----------------------------------------------- + +Instead of requiring the following import:: + + from __future__ import annotations + +the PEP could call the feature more explicitly, for example +``string_annotations``, ``stringify_annotations``, +``annotation_strings``, ``annotations_as_strings``, ``lazy_anotations``, +``static_annotations``, etc. + +The problem with those names is that they are very verbose. Each of +them besides ``lazy_annotations`` would constitute the longest future +feature name in Python. They are long to type and harder to remember +than the single-word form. + +There is precedence of a future import name that sounds overly generic +but in practice was obvious to users as to what it does:: + + from __future__ import division + Prior discussion ================ @@ -538,6 +602,62 @@ helper, which could solve self-referencing classes even in the form of a class decorator. He suggested the PEP should provide this helper function in the standard library. +Second draft discussion on python-dev +------------------------------------- + +Discussion happened mainly in the `announcement thread `_, +followed by a brief discussion under `Mark Shannon's post +`_. + +Steven D'Aprano was concerned whether it's acceptable for typos to be +allowed in annotations after the change proposed by the PEP. Brett +Cannon responded that type checkers and other static analyzers (like +linters or programming text editors) will catch this type of error. +Jukka Lehtosalo added that this situation is analogous to how names in +function bodies are not resolved until the function is called. + +A major topic of discussion was Nick Coghlan's suggestion to store +annotations in "thunk form", in other words as a specialized lambda +which would be able to access class-level scope (and allow for scope +customization at call time). He presented a possible design for it +(`indirect attribute cells +`_). +This was later seen as equivalent to "special forms" in Lisp. Guido van +Rossum expressed worry that this sort of feature cannot be safely +implemented in twelve weeks (i.e. in time before the Python 3.7 beta +freeze). + +After a while it became clear that the point of division between +supporters of the string form vs. supporters of the thunk form is +actually about whether annotations should be perceived as a general +syntactic element vs. something tied to the type checking use case. + +Finally, Guido van Rossum declared he's rejecting the thunk idea +based on the fact that it would require a new building block in the +interpreter. This block would be exposed in annotations, multiplying +possible types of values stored in ``__annotations__`` (arbitrary +objects, strings, and now thunks). Moreover, thunks aren't as +introspectable as strings. Most importantly, Guido van Rossum +explicitly stated interest in gradually restricting the use of +annotations to static typing (with an optional runtime component). + +Nick Coghlan got convinced to PEP 563, too, promptly beginning +the mandatory bike shedding session on the name of the ``__future__`` +import. Many debaters agreed that ``annotations`` seems like +an overly broad name for the feature name. Guido van Rossum briefly +decided to call it ``string_annotations`` but then changed his mind, +arguing that ``division`` is a precedent of a broad name with a clear +meaning. + +The final improvement to the PEP suggested in the discussion by Mark +Shannon was the rejection of the temptation to pass string literals +through to ``__annotations__`` verbatim. + +A side-thread of discussion started around the runtime penalty of +static typing, with topic like the import time of the ``typing`` +module (which is comparable to ``re`` without dependencies, and +three times as heavy as ``re`` when counting dependencies). + Acknowledgements ================