From 877ece9427f0cb5c1b87d1089451d3741f581cd7 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Sun, 10 Sep 2017 20:24:24 +0200 Subject: [PATCH 1/8] PEP 562: module __getattr__ (#412) This is an alternative (or complementary?) proposal to PEP 549. --- pep-562.rst | 105 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 pep-562.rst diff --git a/pep-562.rst b/pep-562.rst new file mode 100644 index 000000000..d16f3b6eb --- /dev/null +++ b/pep-562.rst @@ -0,0 +1,105 @@ +PEP: 562 +Title: Module __getattr__ +Author: Ivan Levkivskyi +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 09-Sep-2017 +Python-Version: 3.7 +Post-History: 09-Sep-2017 + + +Abstract +======== + +It is proposed to support ``__getattr__`` function defined on modules to +provide basic customization of module attribute access. + + +Rationale +========= + +It is sometimes convenient to customize or otherwise have control over +access to module attributes. A typical example is managing deprecation +warnings. Typical workarounds are assigning ``__class__`` of a module object +to a custom subclass of ``types.ModuleType`` or substituting ``sys.modules`` +item with a custom wrapper instance. It would be convenient to simplify this +procedure by recognizing ``__getattr__`` defined directly in a module that +would act like a normal ``__getattr__`` method, except that it will be defined +on module *instances*. For example:: + + # lib.py + + from warnings import warn + + deprecated_names = ["old_function", ...] + + def _deprecated_old_function(arg, other): + ... + + def __getattr__(name): + if name in deprecated_names: + warn(f"{name} is deprecated", DeprecationWarning) + return globals()[f"_deprecated_{name}"] + raise AttributeError(f"module {__name__} has no attribute {name}") + + # main.py + + from lib import old_function # Works, but emits the warning + +There is a related proposal PEP 549 that proposes to support instance +properties for a similar functionality. The difference is this PEP proposes +a faster and simpler mechanism, but provides more basic customization. +An additional motivation for this proposal is that PEP 484 already defines +the use of module ``__getattr__`` for this purpose in Python stub files, +see [1]_. + + +Specification +============= + +The ``__getattr__`` function at the module level should accept one argument +which is a name of an attribute and return the computed value or raise +an ``AttributeError``:: + + def __getattr__(name: str) -> Any: ... + +This function will be called only if ``name`` is not found in the module +through the normal attribute lookup. + +The reference implementation for this PEP can be found in [2]_. + + +Backwards compatibility and impact on performance +================================================= + +This PEP may break code that uses module level (global) name ``__getattr__``. +The performance implications of this PEP are minimal, since ``__getattr__`` +is called only for missing attributes. + + +References +========== + +.. [1] PEP 484 section about ``__getattr__`` in stub files + (https://www.python.org/dev/peps/pep-0484/#stub-files) + +.. [2] The reference implementation + (https://github.com/ilevkivskyi/cpython/pull/3/files) + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: From ae8a94b2feee2a47a0420d9425df8e475a06beda Mon Sep 17 00:00:00 2001 From: Guido van Rossum Date: Sun, 10 Sep 2017 11:26:34 -0700 Subject: [PATCH 2/8] Rename PEP 562 to 0562 (add leading 0 to filename) --- pep-0562.rst | 105 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 pep-0562.rst diff --git a/pep-0562.rst b/pep-0562.rst new file mode 100644 index 000000000..d16f3b6eb --- /dev/null +++ b/pep-0562.rst @@ -0,0 +1,105 @@ +PEP: 562 +Title: Module __getattr__ +Author: Ivan Levkivskyi +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 09-Sep-2017 +Python-Version: 3.7 +Post-History: 09-Sep-2017 + + +Abstract +======== + +It is proposed to support ``__getattr__`` function defined on modules to +provide basic customization of module attribute access. + + +Rationale +========= + +It is sometimes convenient to customize or otherwise have control over +access to module attributes. A typical example is managing deprecation +warnings. Typical workarounds are assigning ``__class__`` of a module object +to a custom subclass of ``types.ModuleType`` or substituting ``sys.modules`` +item with a custom wrapper instance. It would be convenient to simplify this +procedure by recognizing ``__getattr__`` defined directly in a module that +would act like a normal ``__getattr__`` method, except that it will be defined +on module *instances*. For example:: + + # lib.py + + from warnings import warn + + deprecated_names = ["old_function", ...] + + def _deprecated_old_function(arg, other): + ... + + def __getattr__(name): + if name in deprecated_names: + warn(f"{name} is deprecated", DeprecationWarning) + return globals()[f"_deprecated_{name}"] + raise AttributeError(f"module {__name__} has no attribute {name}") + + # main.py + + from lib import old_function # Works, but emits the warning + +There is a related proposal PEP 549 that proposes to support instance +properties for a similar functionality. The difference is this PEP proposes +a faster and simpler mechanism, but provides more basic customization. +An additional motivation for this proposal is that PEP 484 already defines +the use of module ``__getattr__`` for this purpose in Python stub files, +see [1]_. + + +Specification +============= + +The ``__getattr__`` function at the module level should accept one argument +which is a name of an attribute and return the computed value or raise +an ``AttributeError``:: + + def __getattr__(name: str) -> Any: ... + +This function will be called only if ``name`` is not found in the module +through the normal attribute lookup. + +The reference implementation for this PEP can be found in [2]_. + + +Backwards compatibility and impact on performance +================================================= + +This PEP may break code that uses module level (global) name ``__getattr__``. +The performance implications of this PEP are minimal, since ``__getattr__`` +is called only for missing attributes. + + +References +========== + +.. [1] PEP 484 section about ``__getattr__`` in stub files + (https://www.python.org/dev/peps/pep-0484/#stub-files) + +.. [2] The reference implementation + (https://github.com/ilevkivskyi/cpython/pull/3/files) + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: From 051f47153a4ebf89de404fafe664b2afbef9fe29 Mon Sep 17 00:00:00 2001 From: Guido van Rossum Date: Sun, 10 Sep 2017 11:29:01 -0700 Subject: [PATCH 3/8] Fix headers order --- pep-0559.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0559.rst b/pep-0559.rst index f934a0042..4726f3a63 100644 --- a/pep-0559.rst +++ b/pep-0559.rst @@ -6,8 +6,8 @@ Type: Standards Track Content-Type: text/x-rst Created: 2017-09-08 Python-Version: 3.7 -Resolution: https://mail.python.org/pipermail/python-dev/2017-September/149438.html Post-History: 2017-09-09 +Resolution: https://mail.python.org/pipermail/python-dev/2017-September/149438.html Abstract From 1c3d85cc5e38d89b7c3bf319a512df8c28f2076d Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Sun, 10 Sep 2017 20:53:54 +0200 Subject: [PATCH 4/8] PEP 560: Core support for generic types (#411) --- pep-0560.rst | 202 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 202 insertions(+) create mode 100644 pep-0560.rst diff --git a/pep-0560.rst b/pep-0560.rst new file mode 100644 index 000000000..ecc4001eb --- /dev/null +++ b/pep-0560.rst @@ -0,0 +1,202 @@ +PEP: 560 +Title: Core support for generic types +Author: Ivan Levkivskyi +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 03-Sep-2017 +Python-Version: 3.7 +Post-History: 09-Sep-2017 + + +Abstract +======== + +Initially PEP 484 was designed in such way that it would not introduce +*any* changes to the core CPython interpreter. Now type hints and +the ``typing`` module are extensively used by the community, e.g. PEP 526 +and PEP 557 extend the usage of type hints, and the backport of ``typing`` +on PyPI has 1M downloads/moth. Therefore, this restriction can be removed. +It is proposed to add two special methods ``__class_getitem__`` and +``__subclass_base__`` to the core CPython for better support of +generic types. + + +Rationale +========= + +The restriction to not modify the core CPython interpreter lead to some +design decisions that became questionable when the ``typing`` module started +to be widely used. There are three main points of concerns: +performance of the ``typing`` module, metaclass conflicts, and the large +number of hacks currently used in ``typing``. + + +Performance: +------------ + +The ``typing`` module is one of the heaviest and slowest modules in +the standard library even with all the optimizations made. Mainly this is +because subscripted generic types (see PEP 484 for definition of terms +used in this PEP) are class objects (see also [1]_). The three main ways how +the performance can be improved with the help of the proposed special methods: + +- Creation of generic classes is slow since the ``GenericMeta.__new__`` is + very slow; we will not need it anymore. + +- Very long MROs for generic classes will be twice shorter; they are present + because we duplicate the ``collections.abc`` inheritance chain + in ``typing``. + +- Time of instantiation of generic classes will be improved + (this is minor however). + + +Metaclass conflicts: +-------------------- + +All generic types are instances of ``GenericMeta``, so if a user uses +a custom metaclass, then it is hard to make a corresponding class generic. +This is particularly hard for library classes that a user doesn't control. +A workaround is to always mix-in ``GenericMeta``:: + + class AdHocMeta(GenericMeta, LibraryMeta): + pass + + class UserClass(LibraryBase, Generic[T], metaclass=AdHocMeta): + ... + +but this is not always practical or even possible. With the help of the +proposed special attributes the ``GenericMeta`` metaclass will not be needed. + + +Hacks and bugs that will be removed by this proposal: +----------------------------------------------------- + +- ``_generic_new`` hack that exists since ``__init__`` is not called on + instances with a type differing form the type whose ``__new__`` was called, + ``C[int]().__class__ is C``. + +- ``_next_in_mro`` speed hack will be not necessary since subscription will + not create new classes. + +- Ugly ``sys._getframe`` hack, this one is particularly nasty, since it looks + like we can't remove it without changes outside ``typing``. + +- Currently generics do dangerous things with private ABC caches + to fix large memory consumption that grows at least as O(N\ :sup:`2`), + see [2]_. This point is also important because it was recently proposed to + re-implement ``ABCMeta`` in C. + +- Problems with sharing attributes between subscripted generics, + see [3]_. Current solution already uses ``__getattr__`` and ``__setattr__``, + but it is still incomplete, and solving this without the current proposal + will be hard and will need ``__getattribute__``. + +- ``_no_slots_copy`` hack, where we clean-up the class dictionary on every + subscription thus allowing generics with ``__slots__``. + +- General complexity of the ``typing`` module, the new proposal will not + only allow to remove the above mentioned hacks/bugs, but also simplify + the implementation, so that it will be easier to maintain. + + +Specification +============= + +The idea of ``__class_getitem__`` is simple: it is an exact analog of +``__getitem__`` with an exception that it is called on a class that +defines it, not on its instances, this allows us to avoid +``GenericMeta.__getitem__`` for things like ``Iterable[int]``. +The ``__class_getitem__`` is automatically a class method and +does not require ``@classmethod`` decorator (similar to +``__init_subclass__``) and is inherited like normal attributes. +For example:: + + class MyList: + def __getitem__(self, index): + return index + 1 + def __class_getitem__(cls, item): + return f"{cls.__name__}[{item.__name__}]" + + class MyOtherList(MyList): + pass + + assert MyList()[0] == 1 + assert MyList[int] == "MyList[int]" + + assert MyOtherList()[0] == 1 + assert MyOtherList[int] == "MyOtherList[int]" + +Note that this method is used as a fallback, so if a metaclass defines +``__getitem__``, then that will have the priority. + +If an object that is not a class object appears in the bases of a class +definition, the ``__subclass_base__`` is searched on it. If found, +it is called with the original tuple of bases as an argument. If the result +of the call is not ``None``, then it is substituted instead of this object. +Otherwise (if the result is ``None``), the base is just removed. This is +necessary to avoid inconsistent MRO errors, that are currently prevented by +manipulations in ``GenericMeta.__new__``. After creating the class, +the original bases are saved in ``__orig_bases__`` (currently this is also +done by the metaclass). + +NOTE: These two method names are reserved for exclusive use by +the ``typing`` module and the generic types machinery, and any other use is +strongly discouraged. The reference implementation (with tests) can be found +in [4]_, the proposal was originally posted and discussed on +the ``typing`` tracker, see [5]_. + + +Backwards compatibility and impact on users who don't use ``typing``: +===================================================================== + +This proposal may break code that currently uses the names +``__class_getitem__`` and ``__subclass_base__``. + +This proposal will support almost complete backwards compatibility with +the current public generic types API; moreover the ``typing`` module is still +provisional. The only two exceptions are that currently +``issubclass(List[int], List)`` returns True, with this proposal it will raise +``TypeError``. Also ``issubclass(collections.abc.Iterable, typing.Iterable)`` +will return ``False``, which is probably desirable, since currently we have +a (virtual) inheritance cycle between these two classes. + +With the reference implementation I measured negligible performance effects +(under 1% on a micro-benchmark) for regular (non-generic) classes. + + +References +========== + +.. [1] Discussion following Mark Shannon's presentation at Language Summit + (https://github.com/python/typing/issues/432) + +.. [2] Pull Request to implement shared generic ABC caches + (https://github.com/python/typing/pull/383) + +.. [3] An old bug with setting/accessing attributes on generic types + (https://github.com/python/typing/issues/392) + +.. [4] The reference implementation + (https://github.com/ilevkivskyi/cpython/pull/2/files) + +.. [5] Original proposal + (https://github.com/python/typing/issues/468) + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: From 4f82ff5bf3805415658836d212b37911cf029dce Mon Sep 17 00:00:00 2001 From: Guido van Rossum Date: Sun, 10 Sep 2017 12:53:18 -0700 Subject: [PATCH 5/8] Delete wrong file --- pep-562.rst | 105 ---------------------------------------------------- 1 file changed, 105 deletions(-) delete mode 100644 pep-562.rst diff --git a/pep-562.rst b/pep-562.rst deleted file mode 100644 index d16f3b6eb..000000000 --- a/pep-562.rst +++ /dev/null @@ -1,105 +0,0 @@ -PEP: 562 -Title: Module __getattr__ -Author: Ivan Levkivskyi -Status: Draft -Type: Standards Track -Content-Type: text/x-rst -Created: 09-Sep-2017 -Python-Version: 3.7 -Post-History: 09-Sep-2017 - - -Abstract -======== - -It is proposed to support ``__getattr__`` function defined on modules to -provide basic customization of module attribute access. - - -Rationale -========= - -It is sometimes convenient to customize or otherwise have control over -access to module attributes. A typical example is managing deprecation -warnings. Typical workarounds are assigning ``__class__`` of a module object -to a custom subclass of ``types.ModuleType`` or substituting ``sys.modules`` -item with a custom wrapper instance. It would be convenient to simplify this -procedure by recognizing ``__getattr__`` defined directly in a module that -would act like a normal ``__getattr__`` method, except that it will be defined -on module *instances*. For example:: - - # lib.py - - from warnings import warn - - deprecated_names = ["old_function", ...] - - def _deprecated_old_function(arg, other): - ... - - def __getattr__(name): - if name in deprecated_names: - warn(f"{name} is deprecated", DeprecationWarning) - return globals()[f"_deprecated_{name}"] - raise AttributeError(f"module {__name__} has no attribute {name}") - - # main.py - - from lib import old_function # Works, but emits the warning - -There is a related proposal PEP 549 that proposes to support instance -properties for a similar functionality. The difference is this PEP proposes -a faster and simpler mechanism, but provides more basic customization. -An additional motivation for this proposal is that PEP 484 already defines -the use of module ``__getattr__`` for this purpose in Python stub files, -see [1]_. - - -Specification -============= - -The ``__getattr__`` function at the module level should accept one argument -which is a name of an attribute and return the computed value or raise -an ``AttributeError``:: - - def __getattr__(name: str) -> Any: ... - -This function will be called only if ``name`` is not found in the module -through the normal attribute lookup. - -The reference implementation for this PEP can be found in [2]_. - - -Backwards compatibility and impact on performance -================================================= - -This PEP may break code that uses module level (global) name ``__getattr__``. -The performance implications of this PEP are minimal, since ``__getattr__`` -is called only for missing attributes. - - -References -========== - -.. [1] PEP 484 section about ``__getattr__`` in stub files - (https://www.python.org/dev/peps/pep-0484/#stub-files) - -.. [2] The reference implementation - (https://github.com/ilevkivskyi/cpython/pull/3/files) - - -Copyright -========= - -This document has been placed in the public domain. - - - -.. - Local Variables: - mode: indented-text - indent-tabs-mode: nil - sentence-end-double-space: t - fill-column: 70 - coding: utf-8 - End: From 8396bb4fd2f52cc04f6391607704862a94501c15 Mon Sep 17 00:00:00 2001 From: Ethan Smith Date: Sun, 10 Sep 2017 15:34:44 -0700 Subject: [PATCH 6/8] PEP 461: Distributing and Packaging Type Information (#415) --- pep-0561.rst | 179 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 179 insertions(+) create mode 100644 pep-0561.rst diff --git a/pep-0561.rst b/pep-0561.rst new file mode 100644 index 000000000..ebc8158cf --- /dev/null +++ b/pep-0561.rst @@ -0,0 +1,179 @@ +PEP: 561 +Title: Distributing and Packaging Type Information +Author: Ethan Smith +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 09-Sep-2017 +Python-Version: 3.7 +Post-History: + + +Abstract +======== + +PEP 484 introduced type hints to Python, with goals of making typing +gradual and easy to adopt. Currently, typing information must be distributed +manually. This PEP provides a standardized means to package and distribute +type information and an ordering for type checkers to resolve modules and +collect this information for type checking using existing packaging +architecture. + + +Rationale +========= + +PEP 484 has a brief section on distributing typing information. In this +section [1]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for +shipping stub files. However, manually adding a path to stub files for each +third party library does not scale. The simplest approach people have taken +is to add ``site-packages`` to their ``PYTHONPATH``, but this causes type +checkers to fail on packages that are highly dynamic (e.g. sqlalchemy +and Django). + +Furthermore, package authors are wishing to distribute code that has +inline type information, and there currently is no standard method to +distribute packages with inline type annotations or syntax that can +simultaneously be used at runtime and in type checking. + + +Specification +============= + +There are several motivations and methods of supporting typing in a package. This PEP recognizes three (3) types of packages that may be created: + +1. The package maintainer would like to add type information inline. + +2. The package maintainer would like to add type information via stubs. + +3. A third party would like to share stub files for a package, but the + maintainer does not want to include them in the source of the package. + +This PEP aims to support these scenarios and make them simple to add to +packaging and deployment. + +The two major parts of this specification are the packaging specifications +and the resolution order for resolving module type information. This spec +is meant to replace the ``shared/typehints/pythonX.Y/`` spec of PEP 484 [1]_. + +Packaging Type Information +-------------------------- + +Packages must opt into supporting typing. This will be done though a distutils +extension [2]_, providing a ``typed`` keyword argument to the distutils +``setup()`` command. The argument value will depend on the kind of type +information the package provides. The distutils extension will be added to the +``typing`` package. Therefore a package maintainer may write + +:: + + setup( + ... + setup_requires=["typing"], + typed="inline", + ... + ) + +Inline Typed Packages +''''''''''''''''''''' + +Packages that have inline type annotations simply have to pass the value +``"inline"`` to the ``typed`` argument in ``setup()``. + +Stub Only Packages +'''''''''''''''''' + +For package maintainers wishing to ship stub files containing all of their +type information, it is prefered that the ``*.pyi`` stubs are alongside the +corresponding ``*.py`` files. However, the stubs may be put in a sub-folder +of the Python sources, with the same name the ``*.py`` files are in. For +example, the ``flyingcircus`` package would have its stubs in the folder +``flyingcircus/flyingcircus/``. This path is chosen so that if stubs are +not found in ``flyingcircus/`` the type checker may treat the subdirectory as +a normal package. The normal resolution order of checking ``*.pyi`` before +``*.py`` will be maintained. The value of the ``typed`` argument to +``setup()`` is ``"stubs"`` for this type of distribution. The author of the +package is suggested to use ``package_data`` to assure the stub files are +installed alongside the runtime Python code. + +Third Party Stub Packages +''''''''''''''''''''''''' + +Third parties seeking to distribute stub files are encouraged to contact the +maintainer of the package about distribution alongside the package. If the +maintainer does not wish to maintain or package stub files or type information +inline, then a "third party stub package" should be created. The structure is +similar, but slightly different from that of stub only packages. If the stubs +are for the library ``flyingcircus`` then the package should be named +``flyingcircus-stubs`` and the stub files should be put in a sub-directory +named ``flyingcircus``. This allows the stubs to be checked as if they were in +a regular package. These packages should also pass ``"stubs"`` as the value +of ``typed`` argument in ``setup()``. These packages are suggested to use +``package_data`` to package stub files. + +The version of the ``flyingcircus-stubs`` package should match the version of +the ``flyingcircus`` package it is providing types for. + +Type Checker Module Resolution Order +------------------------------------ + +The following is the order that type checkers supporting this PEP should +resolve modules containing type information: + +1. User code - the files the type checker is running on. + +2. Stubs or Python source in ``PYTHONPATH``. This is to allow the user + complete control of which stubs to use, and patch broken stubs/inline + types from packages. + +3. Third party stub packages - these packages can supersede the installed + untyped packages. They can be found at ``pkg-stubs`` for package ``pkg``, + however it is encouraged to check their metadata to confirm that they opt + into type checking. + +4. Inline packages - finally, if there is nothing overriding the installed + package, and it opts into type checking. + +5. Typeshed (if used) - Provides the stdlib types and several third party libraries + +When resolving step (3) type checkers should assure the version of the stubs +match the installed runtime package. + +Type checkers that check a different Python version than the version they run +on must find the type information in the ``site-packages``/``dist-packages`` +of that Python version. This can be queried e.g. +``pythonX.Y -c 'import sys; print(sys.exec_prefix)'``. It is also recommended +that the type checker allow for the user to point to a particular Python +binary, in case it is not in the path. + +To check if a package has opted into type checking, type checkers are +recommended to use the ``pkg_resources`` module to query the package +metadata. If the ``typed`` package metadata has ``None`` as its value, the +package has not opted into type checking, and the type checker should skip that +package. + + +References +========== + +.. [1] PEP 484, Storing and Distributing Stub Files + (https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files) + +.. [2] Distutils Extensions, Adding setup() arguments + (http://setuptools.readthedocs.io/en/latest/setuptools.html#adding-setup-arguments) + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: From 6fe422400aa8ff1dee21d30c0e3331f0c7c881aa Mon Sep 17 00:00:00 2001 From: Nick Coghlan Date: Mon, 11 Sep 2017 13:04:07 +1000 Subject: [PATCH 7/8] PEP 559: Fix typo in my name --- pep-0559.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0559.rst b/pep-0559.rst index 4726f3a63..5206a3a5d 100644 --- a/pep-0559.rst +++ b/pep-0559.rst @@ -61,7 +61,7 @@ you return ``7`` or ``((7,), {})``? And so on. The author claims that you won't ever need the return value of ``noop()`` so it will always return ``None``. -Coghlin's Dialogs (edited for formatting): +Coghlan's Dialogs (edited for formatting): My counterargument to this would be ``map(noop, iterable)``, ``sorted(iterable, key=noop)``, etc. (``filter``, ``max``, and From 454c88950bca978ddff30144c66f2e420f51f96e Mon Sep 17 00:00:00 2001 From: Lukasz Langa Date: Mon, 11 Sep 2017 08:05:32 -0700 Subject: [PATCH 8/8] PEP 563: Postponed Evaluation of Annotations Draft 0. PEP number assigned by GvR in https://github.com/ambv/static-annotations/commit/77557db51c671695fe732135567698b4a4f99b4c#commitcomment-24210467 --- pep-0563.rst | 264 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 264 insertions(+) create mode 100644 pep-0563.rst diff --git a/pep-0563.rst b/pep-0563.rst new file mode 100644 index 000000000..976b32941 --- /dev/null +++ b/pep-0563.rst @@ -0,0 +1,264 @@ +PEP: 563 +Title: Postponed Evaluation of Annotations +Version: $Revision$ +Last-Modified: $Date$ +Author: Łukasz Langa +Discussions-To: Python-Dev +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 8-Sep-2017 +Python-Version: 3.7 +Post-History: +Resolution: + + +Abstract +======== + +PEP 3107 introduced syntax for function annotations, but the semantics +were deliberately left undefined. PEP 484 introduced a standard meaning +to annotations: type hints. PEP 526 defined variable annotations, +explicitly tying them with the type hinting use case. + +This PEP proposes changing function annotations and variable annotations +so that they are no longer evaluated at function definition time. +Instead, they are preserved in ``__annotations__`` in string form. + +This change is going to be introduced gradually, starting with a new +``__future__`` import in Python 3.7. + + +Rationale and Goals +=================== + +PEP 3107 added support for arbitrary annotations on parts of a function +definition. Just like default values, annotations are evaluated at +function definition time. This creates a number of issues for the type +hinting use case: + +* forward references: when a type hint contains names that have not been + defined yet, that definition needs to be expressed as a string + literal; + +* type hints are executed at module import time, which is not + computationally free. + +Postponing the evaluation of annotations solves both problems. + +Non-goals +--------- + +Just like in PEP 484 and PEP 526, it should be emphasized that **Python +will remain a dynamically typed language, and the authors have no desire +to ever make type hints mandatory, even by convention.** + +Annotations are still available for arbitrary use besides type checking. +Using ``@typing.no_type_hints`` in this case is recommended to +disambiguate the use case. + + +Implementation +============== + +In a future version of Python, function and variable annotations will no +longer be evaluated at definition time. Instead, a string form will be +preserved in the respective ``__annotations__`` dictionary. Static type +checkers will see no difference in behavior, whereas tools using +annotations at runtime will have to perform postponed evaluation. + +If an annotation was already a string, this string is preserved +verbatim. In other cases, the string form is obtained from the AST +during the compilation step, which means that the string form preserved +might not preserve the exact formatting of the source. + +Annotations need to be syntactically valid Python expressions, also when +passed as literal strings (i.e. ``compile(literal, '', 'eval')``). +Annotations can only use names present in the module scope as postponed +evaluation using local names is not reliable. + +Note that as per PEP 526, local variable annotations are not evaluated +at all since they are not accessible outside of the function's closure. + +Enabling the future behavior in Python 3.7 +------------------------------------------ + +The functionality described above can be enabled starting from Python +3.7 using the following special import:: + + from __future__ import annotations + + +Resolving Type Hints at Runtime +=============================== + +To resolve an annotation at runtime from its string form to the result +of the enclosed expression, user code needs to evaluate the string. + +For code that uses type hints, the ``typing.get_type_hints()`` function +correctly evaluates expressions back from its string form. Note that +all valid code currently using ``__annotations__`` should already be +doing that since a type annotation can be expressed as a string literal. + +For code which uses annotations for other purposes, a regular +``eval(ann, globals, locals)`` call is enough to resolve the +annotation. The trick here is to get the correct value for globals. +Fortunately, in the case of functions, they hold a reference to globals +in an attribute called ``__globals__``. To get the correct module-level +context to resolve class variables, use:: + + cls_globals = sys.modules[SomeClass.__module__].__dict__ + +Runtime annotation resolution and class decorators +-------------------------------------------------- + +Metaclasses and class decorators that need to resolve annotations for +the current class will fail for annotations that use the name of the +current class. Example:: + + def class_decorator(cls): + annotations = get_type_hints(cls) # raises NameError on 'C' + print(f'Annotations for {cls}: {annotations}') + return cls + + @class_decorator + class C: + singleton: 'C' = None + +This was already true before this PEP. The class decorator acts on +the class before it's assigned a name in the current definition scope. + +The situation is made somewhat stricter when class-level variables are +considered. Previously, when the string form wasn't used in annotations, +a class decorator would be able to cover situations like:: + + @class_decorator + class Restaurant: + class MenuOption(Enum): + SPAM = 1 + EGGS = 2 + + default_menu: List[MenuOption] = [] + +This is no longer possible. + +Runtime annotation resolution and ``TYPE_CHECKING`` +--------------------------------------------------- + +Sometimes there's code that must be seen by a type checker but should +not be executed. For such situations the ``typing`` module defines a +constant, ``TYPE_CHECKING``, that is considered ``True`` during type +checking but ``False`` at runtime. Example:: + + import typing + + if typing.TYPE_CHECKING: + import expensive_mod + + def a_func(arg: expensive_mod.SomeClass) -> None: + a_var: expensive_mod.SomeClass = arg + ... + +This approach is also useful when handling import cycles. + +Trying to resolve annotations of ``a_func`` at runtime using +``typing.get_type_hints()`` will fail since the name ``expensive_mod`` +is not defined (``TYPE_CHECKING`` variable being ``False`` at runtime). +This was already true before this PEP. + + +Backwards Compatibility +======================= + +This is a backwards incompatible change. Applications depending on +arbitrary objects to be directly present in annotations will break +if they are not using ``typing.get_type_hints()`` or ``eval()``. + +Annotations that depend on locals at the time of the function/class +definition are now invalid. Example:: + + def generate_class(): + some_local = datetime.datetime.now() + class C: + field: some_local = 1 # NOTE: INVALID ANNOTATION + def method(self, arg: some_local.day) -> None: # NOTE: INVALID ANNOTATION + ... + +Annotations using nested classes and their respective state are still +valid, provided they use the fully qualified name. Example:: + + class C: + field = 'c_field' + def method(self, arg: C.field) -> None: # this is OK + ... + + class D: + field2 = 'd_field' + def method(self, arg: C.field -> C.D.field2: # this is OK + ... + +In the presence of an annotation that cannot be resolved using the +current module's globals, a NameError is raised at compile time. + + +Deprecation policy +------------------ + +In Python 3.7, a ``__future__`` import is required to use the described +functionality and a ``PendingDeprecationWarning`` is raised by the +compiler in the presence of type annotations in modules without the +``__future__`` import. In Python 3.8 the warning becomes a +``DeprecationWarning``. In the next version this will become the +default behavior. + + +Rejected Ideas +============== + +Keep the ability to use local state when defining annotations +------------------------------------------------------------- + +With postponed evaluation, this is impossible for function locals. For +classes, it would be possible to keep the ability to define annotations +using the local scope. However, when using ``eval()`` to perform the +postponed evaluation, we need to provide the correct globals and locals +to the ``eval()`` call. In the face of nested classes, the routine to +get the effective "globals" at definition time would have to look +something like this:: + + def get_class_globals(cls): + result = {} + result.update(sys.modules[cls.__module__].__dict__) + for child in cls.__qualname__.split('.'): + result.update(result[child].__dict__) + return result + +This is brittle and doesn't even cover slots. Requiring the use of +module-level names simplifies runtime evaluation and provides the +"one obvious way" to read annotations. It's the equivalent of absolute +imports. + + +Acknowledgements +================ + +This document could not be completed without valuable input, +encouragement and advice from Guido van Rossum, Jukka Lehtosalo, and +Ivan Levkivskyi. + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: