diff --git a/pep-0526.txt b/pep-0526.txt index 7413996ee..5fbbb30a8 100644 --- a/pep-0526.txt +++ b/pep-0526.txt @@ -2,53 +2,66 @@ PEP: 526 Title: Syntax for Variable and Attribute Annotations Version: $Revision$ Last-Modified: $Date$ -Author: Ryan Gonzalez , Philip House , Guido van Rossum +Author: Ryan Gonzalez , Philip House , Ivan Levkivskyi , Lisa Roach , Guido van Rossum Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Aug-2016 Python-Version: 3.6 + Notice for Reviewers ==================== -This PEP is not ready for review. We're merely committing changes -frequently so we don't end up with a huge merge conflict. For minor -textual nits please use https://github.com/python/peps/pull/72. For -discussion about contents, please refer to -https://github.com/python/typing/issues/258 (but please be patient, we -know we're way behind addressing all comments). +This PEP was drafted in a separate repo: +https://github.com/phouse512/peps/tree/pep-0526. + +There was preliminary discussion on python-ideas and at +https://github.com/python/typing/issues/258. + +Before you bring up an objection in a public forum please at least +read the summary of rejected ideas listed at the end of this PEP. + Abstract ======== -PEP 484 introduced type hints and; In particular, it introduced the notion of -type comments:: +PEP 484 introduced type hints, a.k.a. type annotations. While its +main focus was function annotations, it also introduced the notion of +type comments to annotate variables:: - # a is specified to be a list of ints. - a = [] # type: List[int] - # b is a string - b = None # type: str - class Cls: - my_class_attr = True # type: bool + # 'primes' is a list of integers + primes = [] # type: List[int] -This PEP aims at adding syntax to Python for annotating the types of variables and -attributes, instead of expressing them through comments:: + # 'captain' is a string (Note: initial value is a problem) + captain = ... # type: str + + class Starship: + # 'stats' is a class attribute + stats = {} # type: Dict[str, int] + +This PEP aims at adding syntax to Python for annotating the types of variables +and attributes, instead of expressing them through comments:: + + primes: List[int] = [] + + captain: str # Note: no initial value! + + class Starship: + stats: ClassVar[Dict[str, int]] = {} - a: List[int] = [] - b: str - class Cls: - my_class_attr: ClassAttr[bool] = True Rationale ========= -Although type comments work well, the fact that they're expressed through -comments has some downsides: +Although type comments work well enough, the fact that they're +expressed through comments has some downsides: - Text editors often highlight comments differently from type annotations. -- There isn't a way to annotate the type of an undefined variable; you need to + +- There's no way to annotate the type of an undefined variable; you need to initialize it to ``None`` (e.g. ``a = None # type: int``). + - Variables annotated in a conditional branch are difficult to read:: if some_value: @@ -59,21 +72,54 @@ comments has some downsides: - Since type comments aren't actually part of the language, if a Python script wants to parse them, it would require a custom parser instead of just using ``ast``. -- It's impossible to retrieve the annotations at runtime outside of attempting to - find the module's source code and parse it at runtime, which is inelegant, to - say the least. -The majority of these issues can be alleviated by making the syntax a core part of -the language. +- Type comments are used a lot in typeshed. Migrating typeshed to use + the variable annotation syntax instead of type comments would improve + readability of stubs. + +- In situations where normal comments and type comments used together, it is + difficult to distinguish them:: + + path = None # type: Optional[str] # Path to module source + +- It's impossible to retrieve the annotations at runtime outside of + attempting to find the module's source code and parse it at runtime, + which is inelegant, to say the least. + +The majority of these issues can be alleviated by making the syntax +a core part of the language. + +Non-goals +********* + +While the proposal is accompanied by an extension of ``typing.get_type_hints`` +standard library function for runtime retrieval of annotations, the variable +annotations are not designed for runtime type checking. Third party packages +would have to be developed to implement such functionality. + +It should also be emphasized that **Python will remain a dynamically typed +language, and the authors have no desire to ever make type hints mandatory, +even by convention.** The goal of annotation syntax is to provide an +easy way to specify the structured type metadata for third party tools. + Specification ============= -*** big key concepts, not quite sure what the best way to organize this would be, -or if they deserve their own sections *** +Type annotation can be added to an assignment statement or to a simple +name indicating the desired type of the annotation target to a third +party type checker:: -Normal Variable Typing -********************** + my_var: int + my_var = 5 # Passes type check. + other_var: int = 'a' # Flagged as error by type checker, + # but OK at runtime. + +Below we specify the semantics of type annotations for type checkers +in different contexts and their runtime effects. + +Variable Annotations +******************** The types of locals and globals can be annotated as follows:: @@ -89,20 +135,17 @@ assigned in conditional branches:: else: sane_world = False -Note that, although this syntax does allow tuple packing, it does *not* allow one -to annotate the types of variables when tuple unpacking is used:: +Note that, although the syntax does allow tuple packing, it does *not* allow +one to annotate the types of variables when tuple unpacking is used:: # Tuple packing with variable annotation syntax - t: Any = (1, 2, 3) - - # Tuple unpacking with type comments - x, y, z = t # type: int, int, int + t: Tuple[int, ...] = (1, 2, 3) # Tuple unpacking with variable annotation syntax - x: int - y: int - z: int - x, y, z = t + header: str + kind: int + body: Optional[List[str]] + header, kind, body = message Omitting a default value leaves the variable uninitialized:: @@ -115,7 +158,7 @@ it a local:: def f(): a: int print(a) # raises UnboundLocalError - # Commenting out the `a: int` makes it a NameError! + # Commenting out the ``a: int`` makes it a NameError. as if the code were:: @@ -123,97 +166,384 @@ as if the code were:: if False: a = 0 print(a) # raises UnboundLocalError - -Class Variable Typing -********************* - -Adding variable types allow for us annotate the types of instance variables in class -bodies. In particular, the value-less notation (`a: int`) allows us to annotate -instance variables that should be initialized in `__init__` or `__new__`. The -proposed syntax looks as follows:: - - class Starship: - captain: str # instance variable without default - damage: int = 0 # instance variable with default - stats: class Dict[str, int] = {} # class variable with initialization - - -Duplicate annotations -********************* - -Any duplicate type annotations will be ignored:: +Duplicate type annotations will be ignored. However, static type +checkers will issue a warning for annotations of the same variable +by a different type:: a: int - a: int # Doesn't do anything. + a: str # Static type checker will warn about this. -The Python compiler will not validate the type expression, and leave it to -the type checker to complain. The above code will be allowed by the -compiler at runtime. +``__annotations__`` is writable, so this is permitted:: + + __annotations__['s'] = str + +But attempting to update ``__annotations__`` to something other than a dict +may result in a TypeError:: + + class C: + __annotations__ = 42 + x: int = 5 # raises TypeError + +(Note that the assignment to ``__annotations__``, which is the +culprit, is accepted by the Python interpreter without questioning it +-- but the subsequent type annotation expects it to be a +``MutableMapping`` and will fail.) + + +Attribute annotations +********************* + +Type annotations can also be used to annotate attributes +in class bodies. In particular, the value-less notation ``a: int`` allows us +to annotate instance variables that should be initialized in ``__init__`` +or ``__new__``. The proposed syntax is as follows:: + + class BasicStarship: + captain: str = 'Picard' # instance variable with default + damage: int # instance variable without default + stats: ClassVar[Dict[str, int]] = {} # class variable + +Here ``ClassVar`` is a special class in typing module that indicates to +static type checker that this attribute should not be set on class instances. +This could be illustrated with a more detailed example. In this class:: + + class Starship: + captain = 'Picard' + stats = {} + def __init__(self, damage, captain=None): + self.damage = damage + if captain: + self.captain = captain # Else keep the default + def hit(self): + Starship.stats['hits'] = Starship.stats.get('hits', 0) + 1 + +``stats`` is intended to be a class variable (keeping track of many different +per-game statistics), while ``captain`` is an instance variable with a default +value set in the class. This difference could not be seen by type +checker -- both get initialized in the class, but ``captain`` serves only +as a convenient default value for the instance variable, while ``stats`` +is truly a class variable -- it is intended to be shared by all instances. + +Since both variables happen to be initialized at the class level, it is +useful to distinguish them by marking class variables as annotated with +types wrapped in ``ClassVar[...]``. In such way type checker will prevent +accidental assignments to attributes with a same name on class instances. +For example, annotating the discussed class:: + + class Starship: + captain: str = 'Picard' + damage: int + stats: ClassVar[Dict[str, int]] = {} + def __init__(self, damage: int, captain: str = None): + self.damage = damage + if captain: + self.captain = captain # Else keep the default + def hit(self): + Starship.stats['hits'] = Starship.stats.get('hits', 0) + 1 + + enterprise_d = Starship(3000) + enterprise_d.stats = {} # Flagged as error by a type checker + Starship.stats = {} # This is OK + +As a matter of convenience, instance attributes can be annotated in +``__init__`` or other methods, rather than in class:: + + from typing import Generic, TypeVar + T = TypeVar(’T’) + + class Box(Generic[T]): + def __init__(self, content): + self.content: T = content + +Annotating expressions +********************** + +If the initial value is specified, then the target of the annotation can be +any valid single assignment target:: + + class Cls: + pass + + c = Cls() + c.x: int = 0 # Annotates c.x with int. + c.y: int # Invalid syntax: no initial value was specified! + + d = {} + d['a']: int = 0 # Annotates d['a'] with int. + d['b']: int # Invalid again. + +Note that even ``(my_var)`` is considered an expression, not a simple name. +Consequently:: + + (x): int # Invalid syntax + (x): int = 0 # OK + +It is up to the type checker to decide exactly when to accept this syntax. Where annotations aren't allowed ******************************** -It's illegal to attempt to annotate ``global`` and ``nonlocal``:: +It is illegal to attempt to annotate variables subject to ``global`` +or ``nonlocal`` in the same function scope:: def f(): global x: int # SyntaxError + def g(): + x: int # Also a SyntaxError + global x + The reason is that ``global`` and ``nonlocal`` don't own variables; therefore, the type annotations belong in the scope owning the variable. -In addition, you cannot annotate variable used in a ``for`` or ``with`` -statement; they must be annotated ahead of time, in a similar manner to tuple +Only single assignment targets and single right hand side values are allowed. +In addition, one cannot annotate variables used in a ``for`` or ``with`` +statement; they can be annotated ahead of time, in a similar manner to tuple unpacking:: a: int for a in my_iter: - f: MyFile - with myfunc() as f: - # ... + ... -Capturing Types at Runtime -************************** + f: MyFile + with myfunc() as f: + ... -In order to capture variable types that are usable at runtime, we store the -types in `__annotations__` as dictionaries at various levels. At each level (for -example, global), the types dictionary would be stored in the `__annotations__` -dictionary for that given level. Here is an example for both global and class -level types:: - # print global type annotations +Changes to standard library and documentation +============================================= + +- A new covariant type ``ClassVar[T_co]`` is added to the ``typing`` + module. It accepts only a single argument that should be a valid type, + and is used to annotate class variables that should no be set on class + instances. This restriction is ensured by static checkers, + but not at runtime. See Attribute Annotations for examples + and explanations for the usage of ``ClassVar``, and see the Rejected + Proposals section for more information on the reasoning behind ``ClassVar``. + +- Function ``get_type_hints`` in the ``typing`` module will be extended, + so that one can retrieve type annotations at runtime from modules + and classes in addition to functions. + Annotations are returned as a dictionary mapping from variable, arguments, + or attributes to their type hints with forward references evaluated. + For classes it returns a mapping (perhaps ``collections.ChainMap``) + constructed from annotations in method resolution order. + +- Recommended guidelines for using annotations will be added to the + documentation, containing a pedagogical recapitulation of specifications + described in this PEP and in PEP 484. In addition, a helper script for + translating type comments into type annotations will be published + separately from the standard library. + + +Runtime effects of type annotations +=================================== + +Annotating a local variable will cause +the interpreter to treat it as a local, even if it was never assigned to. +Annotations for local variables will not be evaluated:: + + def f(): + x: NonexistentName # No error. + +However, if it is at a module or class level, then the type *will* be +evaluated:: + + x: NonexistentName # Error! + class X: + attr: NonexistentName # Error! + +In addition, at the module or class level, if the item being annotated is a +simple name, then it and the annotation will be stored in the +``__annotations__`` attribute of that module or class as a dictionary mapping +from names to evaluated annotations. Here is an example:: + + from typing import Dict + class Player: + ... players: Dict[str, Player] + print(__annotations__) + # prints: {'players': typing.Dict[str, __main__.Player]} - # print class type annotations +The recommended way of getting annotations at runtime is by using +``typing.get_type_hints`` function; as with all dunder attributes, +any undocummented use of ``__annotations__`` is subject to breakage +without warning:: + + from typing import Dict, ClassVar, get_type_hints class Starship: - hitpoints: class int = 50 - stats: class Dict[str, int] = {} - shield: int = 100 - captain: str # no initial value - print(Starship.__annotations__) + hitpoints: int = 50 + stats: ClassVar[Dict[str, int]] = {} + shield: int = 100 + captain: str + def __init__(self, captain: str) -> None: + ... -A note about locals -- the value of having annotations available locally does not -offset the cost of having to create and populate the annotations dictionary on -every function call. + assert get_type_hints(Starship) == {'hitpoints': int, + 'stats': ClassVar[Dict[str, int]], + 'shield': int, + 'captain': str} -These annotations would be printed out from the previous program as follows:: + assert get_type_hints(Starship.__init__) == {'captain': str, + 'return': None} - {'players': Dict[str, Player]} +Note that if annotations are not found statically, then the +``__annotations__`` dictionary is not created at all. Also the +value of having annotations available locally does not offset +the cost of having to create and populate the annotations dictionary +on every function call. Therefore annotations at function level are +not evaluated and not stored. - {'hitpoints': ClassVar[int], - 'stats': ClassVar[Dict[str, int]], - 'shield': int, - 'captain': str - } +Other uses of annotations +************************* + +While Python with this PEP will not object to:: + + alice: 'well done' = 'A+' + bob: 'what a shame' = 'F-' + +since it will not care about the type annotation beyond "it evaluates +without raising", a type checker that encounters it will flag it, +unless disabled with ``# type: ignore`` or ``@no_type_check``. + +However, since Python won't care what the "type" is, +if the above snippet is at the global level or in a class, ``__annotations__`` +will include ``{'alice': 'well done', 'bob': 'what a shame'}``. + +These stored annotations might be used for other purposes, +but with this PEP we explicitly recommend type hinting as the +preferred use of annotations. + + +Rejected proposals and things left out for now +============================================== + +- **Should we introduce variable annotations at all?** + Variable annotations have *already* been around for almost two years + in the form of type comments, sanctioned by PEP 484. They are + extensively used by third party type checkers (mypy, pytype, + PyCharm, etc.) and by projects using the type checkers. However, the + comment syntax has many downsides listed in Rationale. This PEP is + not about the need for type annotations, it is about what should be + the syntax for such annotations. + +- **Introduce a new keyword:** + The choice of a good keyword is hard, + e.g. it can't be ``var`` because that is way too common a variable name, + and it can't be ``local`` if we want to use it for class variables or + globals. Second, no matter what we choose, we'd still need + a ``__future__`` import. + +- **Allow type annotations for tuple unpacking:** + + This cause an ambiguity: It's not clear What meaning should be + assigned to this statement:: + + x, y: T + + Are ``x`` and ``y`` both of type ``T``, or do we expect ``T`` to be + a tuple type of two items that are distributed over ``x`` and ``y``, + or perhaps ``x`` has type ``Any`` and ``y`` has type ``T``? (The + latter is what this would mean if this occurred in a function + signature.) Rather than leave the (human) reader guessing, we + forbid this, at least for now. + +- **Parenthesized form ``(var: type)`` for annotations:** + It was brought up on python-ideas as a remedy for the above-mentioned + ambiguity, but it was rejected since such syntax would be hairy, + the benefits are slight, and the readability would be poor. + +- **Allow annotations in chained assignments:** + This has problems of ambiguity and readability similar to tuple + unpacking, for example in:: + + x: int = y = 1 + z = w: int = 1 + + it is ambiguous, what should be the type of ``y``, and what should + be the type of ``z``. Also the second line is difficult to parse. + +- **Allow annotations in ``with`` and ``for`` statement:** + This was rejected because in ``for`` it would make it hard to spot the actual + iterable, and in ``with`` it would confuse the CPython's LL(1) parser. + +- **Evaluate local annotations at function definition time:** + This has been rejected by Guido because the placement of the annotation + strongly suggests that it's in the same scope as the surrounding code. + +- **Store variable annotations also in function scope:** + The value of having the annotations available locally is just not enough + to significantly offset the cost of creating and populating the dictionary + on *each* function call. + +- **Initialize variables annotated without assignment:** + It was proposed on python-ideas to initialize ``x`` in ``x: int`` to + ``None`` or to an additional special constant like Javascript's + ``undefined``. However, adding yet another singleton value to the language + would needed to be checked for everywhere in the code. Therefore, + Guido just said plain "No" to this. + +- **Add also** ``InstanceAttr`` **to the typing module:** + This is redundant because instance variables are way more common than + class variables. The more common usage deserves to be the default. + +- **Allow instance attribute annotations only in methods:** + The problem is that many ``__init__`` methods do a lot of things besides + initializing instance variables, and it would be harder (for a human) + to find all the instance variable declarations. + And sometimes ``__init__`` is factored into more helper methods + so it's even harder to chase them down. Putting the instance variable + declarations together in the class makes it easier to find them, + and helps a first-time reader of the code. + +- **Use syntax** ``x: class t = v`` **for class variables:** + This would require a more complicated parser and the ``class`` + keyword would confuse simple-minded syntax highlighters. Anyway we + need to have ``ClassVar`` to store class variables to + ``__annotations__``, so that it was decided to go with a simpler + syntax. + +- **Forget about** ``ClassVar`` **altogether:** + This was proposed since mypy seems to be getting along fine without a way + to distinguish between class and instance variables. But a type checker + can do useful things with the extra information, for example flag + accidental assignments to a class variable via the instance + (which would create an instance variable shadowing the class variable). + It could also flag instance variables with mutable defaults, + a well-known hazard. + +- **Do not evaluate annotations, treat them as strings:** + This would be inconsistent with the behavior of function annotations that + are always evaluated. Although this might be reconsidered in future, + it was decided in PEP 484 that this would have to be a separate PEP. + +- **Declare attribute types in class docstring:** + Many projects already use various docstring conventions, often without + much consistency and generally without conforming to the PEP 484 annotation + syntax yet. Also this would require a special sophisticated parser. + This, in turn, would defeat the purpose of the PEP -- + collaborating with the third party type checking tools. + +- **Implement ``__annotations__`` as a descriptor:** + This was proposed to prohibit setting ``__annotations__`` to something + non-dictionary or non-None. Guido has rejected this idea as unnecessary; + instead a TypeError will be raised if an attempt is made to update + ``__annotations__`` when it is anything other than a dict. -Mypy supports allowing `# type` on assignments to instance variables and other things. -In case you prefer annotating instance variables in `__init__` or `__new__`, you can -also annotate variable types for instance variables in methods. Despite this, -`__annotations__` will not be updated for that class. Backwards Compatibility ======================= +This PEP is fully backwards compatible. + + +Implementation +============== + +An implementation for Python 3.6 is found on GitHub repo at +https://github.com/ilevkivskyi/cpython/tree/pep-526 + Copyright =========