pep-0492: v4.

pep-0492.txt

@@ -8,7 +8,7 @@ Type: Standards Track
 Content-Type: text/x-rst
 Created: 09-Apr-2015
 Python-Version: 3.5
-Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015
+Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015
 
 
 Abstract
@@ -57,8 +57,7 @@ Specification
 =============
 
 This proposal introduces new syntax and semantics to enhance coroutine
-support in Python, it does not change the internal implementation of
+support in Python.
-coroutines, which are still based on generators.
 
 This specification presumes knowledge of the implementation of
 coroutines in Python (PEP 342 and PEP 380).  Motivation for the syntax
@@ -66,21 +65,22 @@ changes proposed here comes from the asyncio framework (PEP 3156) and
 the "Cofunctions" proposal (PEP 3152, now rejected in favor of this
 specification).
 
-From this point in this document we use the word *coroutine* to refer
+From this point in this document we use the word *native coroutine* to
-to functions declared using the new syntax.  *generator-based
+refer to functions declared using the new syntax.  *generator-based
 coroutine* is used where necessary to refer to coroutines that are
-based on generator syntax.
+based on generator syntax.  *coroutine* is used in contexts where both
+definitions are applicable.
 
 
 New Coroutine Declaration Syntax
 --------------------------------
 
-The following new syntax is used to declare a coroutine::
+The following new syntax is used to declare a *native coroutine*::
 
     async def read_data(db):
         pass
 
-Key properties of coroutines:
+Key properties of *native coroutines*:
 
 * ``async def`` functions are always coroutines, even if they do not
   contain ``await`` expressions.
@@ -88,10 +88,16 @@ Key properties of coroutines:
 * It is a ``SyntaxError`` to have ``yield`` or ``yield from``
   expressions in an ``async`` function.
 
-* Internally, a new code object flag - ``CO_COROUTINE`` - is introduced
+* Internally, two new code object flags were introduced:
-  to enable runtime detection of coroutines (and migrating existing
+
-  code). All coroutines have both ``CO_COROUTINE`` and ``CO_GENERATOR``
+  - ``CO_COROUTINE`` is used to enable runtime detection of
-  flags set.
+    *coroutines* (and migrating existing code).
+
+  - ``CO_NATIVE_COROUTINE`` is used to mark *native coroutines*
+    (defined with new syntax.)
+
+  All coroutines have ``CO_COROUTINE``, ``CO_NATIVE_COROUTINE``, and
+  ``CO_GENERATOR`` flags set.
 
 * Regular generators, when called, return a *generator object*;
   similarly, coroutines return a *coroutine object*.
@@ -100,15 +106,25 @@ Key properties of coroutines:
   and are replaced with a ``RuntimeError``.  For regular generators
   such behavior requires a future import (see PEP 479).
 
+* See also `Coroutine objects`_ section.
+
 
 types.coroutine()
 -----------------
 
 A new function ``coroutine(gen)`` is added to the ``types`` module.  It
-applies ``CO_COROUTINE`` flag to the passed generator-function's code
+allows interoperability between existing *generator-based coroutines*
-object, making it to return a *coroutine object* when called.
+in asyncio and *native coroutines* introduced by this PEP.
 
-This feature enables an easy upgrade path for existing libraries.
+The function applies ``CO_COROUTINE`` flag to generator-function's code
+object, making it return a *coroutine object*.
+
+The function can be used as a decorator, since it modifies generator-
+functions in-place and returns them.
+
+Note, that the ``CO_NATIVE_COROUTINE`` flag is not applied by
+``types.coroutine()`` to make it possible to separate *native
+coroutines* defined with new syntax, from *generator-based coroutines*.
 
 
 Await Expression
@@ -129,7 +145,9 @@ It uses the ``yield from`` implementation with an extra step of
 validating its argument.  ``await`` only accepts an *awaitable*, which
 can be one of:
 
-* A *coroutine object* returned from a *coroutine* or a generator
+* A *native coroutine object* returned from a *native coroutine*.
+
+* A *generator-based coroutine object* returned from a generator
   decorated with ``types.coroutine()``.
 
 * An object with an ``__await__`` method returning an iterator.
@@ -142,7 +160,7 @@ can be one of:
   explanation.)
 
   To enable this behavior for coroutines, a new magic method called
-  ``__await__`` is added.  In asyncio, for instance, to enable Future
+  ``__await__`` is added.  In asyncio, for instance, to enable *Future*
   objects in ``await`` statements, the only change is to add
   ``__await__ = __iter__`` line to ``asyncio.Future`` class.
 
@@ -160,7 +178,9 @@ can be one of:
 * Objects defined with CPython C API with a ``tp_await`` function,
   returning an iterator (similar to ``__await__`` method).
 
-It is a ``SyntaxError`` to use ``await`` outside of a coroutine.
+It is a ``SyntaxError`` to use ``await`` outside of an ``async def``
+function (like it is a ``SyntaxError`` to use ``yield`` outside of
+``def`` function.)
 
 It is a ``TypeError`` to pass anything other than an *awaitable* object
 to an ``await`` expression.
@@ -169,11 +189,22 @@ to an ``await`` expression.
 Updated operator precedence table
 '''''''''''''''''''''''''''''''''
 
-``await`` keyword is defined differently from ``yield`` and ``yield
+``await`` keyword is defined as follows::
-from`` in the Grammar.
 
-The key difference is that *await expressions* do not require
+    power ::=  await ["**" u_expr]
-parentheses around them most of the times.
+    await ::=  ["await"] primary
+
+where "primary" represents the most tightly bound operations of the
+language.  Its syntax is::
+
+    primary ::=  atom | attributeref | subscription | slicing | call
+
+See Python Documentation [12]_ and `Grammar Updates`_ section of this
+proposal for details.
+
+The key ``await`` difference from ``yield`` and ``yield from``
+operators is that *await expressions* do not require parentheses around
+them most of the times.
 
 Also, ``yield from`` allows any expression as its argument, including
 expressions like ``yield from a() + b()``, that would be parsed as
@@ -186,7 +217,8 @@ operators.
 +------------------------------+-----------------------------------+
 | Operator                     | Description                       |
 +==============================+===================================+
-| ``yield``, ``yield from``    | Yield expression                  |
+| ``yield`` ``x``,             | Yield expression                  |
+| ``yield from`` ``x``         |                                   |
 +------------------------------+-----------------------------------+
 | ``lambda``                   | Lambda expression                 |
 +------------------------------+-----------------------------------+
@@ -221,7 +253,7 @@ operators.
 +------------------------------+-----------------------------------+
 | ``**``                       | Exponentiation                    |
 +------------------------------+-----------------------------------+
-| ``await``                    | Await expression                  |
+| ``await`` ``x``              | Await expression                  |
 +------------------------------+-----------------------------------+
 | ``x[index]``,                | Subscription, slicing,            |
 | ``x[index:index]``,          | call, attribute reference         |
@@ -234,8 +266,6 @@ operators.
 | ``{expressions...}``         | set display                       |
 +------------------------------+-----------------------------------+
 
-See `Grammar Updates`_ section for details.
-
 
 Examples of "await" expressions
 '''''''''''''''''''''''''''''''
@@ -266,8 +296,6 @@ Expression                         Should be written as
 ``await -coro()``                  ``await (-coro())``
 ================================== ==================================
 
-See `Grammar Updates`_ section for details.
-
 
 Asynchronous Context Managers and "async with"
 ----------------------------------------------
@@ -306,18 +334,13 @@ which is semantically equivalent to::
     exc = True
 
     try:
-        try:
+        VAR = await aenter
-            VAR = await aenter
+        BLOCK
-            BLOCK
+    except:
-        except:
+        if not await aexit(mgr, *sys.exc_info()):
-            exc = False
+            raise
-            exit_res = await aexit(mgr, *sys.exc_info())
+    else:
-            if not exit_res:
+        await aexit(mgr, None, None, None)
-                raise
-
-    finally:
-        if exc:
-            await aexit(mgr, None, None, None)
 
 
 As with regular ``with`` statements, it is possible to specify multiple
@@ -325,13 +348,13 @@ context managers in a single ``async with`` statement.
 
 It is an error to pass a regular context manager without ``__aenter__``
 and ``__aexit__`` methods to ``async with``.  It is a ``SyntaxError``
-to use ``async with`` outside of a coroutine.
+to use ``async with`` outside of an ``async def`` function.
 
 
 Example
 '''''''
 
-With asynchronous context managers it is easy to implement proper
+With *asynchronous context managers* it is easy to implement proper
 database transaction managers for coroutines::
 
     async def commit(session, data):
@@ -416,7 +439,7 @@ which is semantically equivalent to::
 
 It is a ``TypeError`` to pass a regular iterable without ``__aiter__``
 method to ``async for``.  It is a ``SyntaxError`` to use ``async for``
-outside of a coroutine.
+outside of an ``async def`` function.
 
 As for with regular ``for`` statement, ``async for`` has an optional
 ``else`` clause.
@@ -536,11 +559,61 @@ Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions
 raised in coroutines are wrapped in ``RuntimeError``.
 
 
+Coroutine objects
+-----------------
+
+Differences from generators
+'''''''''''''''''''''''''''
+
+This section applies only to *native coroutines* with
+``CO_NATIVE_COROUTINE`` flag, i.e. defined with the new ``async def``
+syntax.
+
+**The behavior of existing *generator-based coroutines* in asyncio
+remains unchanged.**
+
+Great effort has been made to make sure that coroutines and
+generators are treated as distinct concepts:
+
+1. *Native coroutine objects* do not implement ``__iter__`` and
+   ``__next__`` methods.  Therefore, they cannot be iterated over or
+   passed to ``iter()``, ``list()``, ``tuple()`` and other built-ins.
+   They also cannot be used in a ``for..in`` loop.
+
+   An attempt to use ``__iter__`` or ``__next__`` on a *native
+   coroutine object* will result in a ``TypeError``.
+
+2. *Plain generators* cannot ``yield from`` *native coroutine objects*:
+   doing so will result in a ``TypeError``.
+
+3. *generator-based coroutines* (for asyncio code must be decorated
+   with ``@asyncio.coroutine``) can ``yield from`` *native coroutine
+   objects*.
+
+4. ``inspect.isgenerator()`` and ``inspect.isgeneratorfunction()``
+   return ``False`` for *native coroutine objects* and *native
+   coroutine functions*.
+
+
+Coroutine object methods
+''''''''''''''''''''''''
+
+Coroutines are based on generators internally, thus they share the
+implementation.  Similarly to generator objects, coroutine objects have
+``throw()``, ``send()`` and ``close()`` methods.  ``StopIteration`` and
+``GeneratorExit`` play the same role for coroutine objects (although
+PEP 479 is enabled by default for coroutines).  See PEP 342, PEP 380,
+and Python Documentation [11]_ for details.
+
+``throw()``, ``send()`` methods for coroutine objects are used to push
+values and raise errors into *Future-like* objects.
+
+
 Debugging Features
 ------------------
 
-One of the most frequent mistakes that people make when using
+A common beginner mistake is forgetting to use ``yield from`` on
-generators as coroutines is forgetting to use ``yield from``::
+coroutines::
 
     @asyncio.coroutine
     def useful():
@@ -590,65 +663,58 @@ Example::
                                     #    previously set wrapper
     assert not isinstance(debug_me(), asyncio.CoroWrapper)
 
-If ``sys.set_coroutine_wrapper()`` is called twice, the new wrapper
-replaces the previous wrapper.  ``sys.set_coroutine_wrapper(None)``
-unsets the wrapper.
 
+New Standard Library Functions
+------------------------------
 
-inspect.iscoroutine() and inspect.iscoroutineobject()
+* ``types.coroutine(gen)``.  See `types.coroutine()`_ section for
------------------------------------------------------
+  details.
-
-Two new functions are added to the ``inspect`` module:
 
 * ``inspect.iscoroutine(obj)`` returns ``True`` if ``obj`` is a
-  coroutine object.
+  *coroutine object*.
 
-* ``inspect.iscoroutinefunction(obj)`` returns ``True`` is ``obj`` is a
+* ``inspect.iscoroutinefunction(obj)`` returns ``True`` if ``obj`` is a
-  coroutine function.
+  *coroutine function*.
 
+* ``inspect.isawaitable(obj)`` returns ``True`` if ``obj`` can be used
+  in ``await`` expression.  See `Await Expression`_ for details.
 
-Differences between coroutines and generators
+* ``sys.set_coroutine_wrapper(wraper)`` allows to intercept creation of
----------------------------------------------
+  *coroutine objects*.  ``wraper`` must be a callable that accepts one
+  argument: a *coroutine object* or ``None``.  ``None`` resets the
+  wrapper.  If called twice, the new wrapper replaces the previous one.
+  See `Debugging Features`_ for more details.
 
-A great effort has been made to make sure that coroutines and
+* ``sys.get_coroutine_wrapper()`` returns the current wrapper object.
-generators are separate concepts:
+  Returns ``None`` if no wrapper was set. See  `Debugging Features`_
-
+  for more details.
-1. Coroutine objects do not implement ``__iter__`` and ``__next__``
-   methods.  Therefore they cannot be iterated over or passed to
-   ``iter()``, ``list()``, ``tuple()`` and other built-ins.  They
-   also cannot be used in a ``for..in`` loop.
-
-2. ``yield from`` does not accept coroutine objects (unless it is used
-   in a generator-based coroutine decorated with ``types.coroutine``.)
-
-3. ``yield from`` does not accept coroutine objects from plain Python
-   generators (*not* generator-based coroutines.)
-
-4. ``inspect.isgenerator()`` and ``inspect.isgeneratorfunction()``
-   return ``False`` for coroutine objects and coroutine functions.
-
-
-Coroutine objects
------------------
-
-Coroutines are based on generators internally, thus they share the
-implementation.  Similarly to generator objects, coroutine objects have
-``throw``, ``send`` and ``close`` methods.  ``StopIteration`` and
-``GeneratorExit`` play the same role for coroutine objects (although
-PEP 479 is enabled by default for coroutines).
 
 
 Glossary
 ========
 
+:Native coroutine:
+    A coroutine function is declared with ``async def``. It uses
+    ``await`` and ``return value``; see `New Coroutine Declaration
+    Syntax`_ for details.
+
+:Native coroutine object:
+    Returned from a native coroutine function. See `Await Expression`_
+    for details.
+
+:Generator-based coroutine:
+    Coroutines based on generator syntax.  Most common example are
+    functions decorated with ``@asyncio.coroutine``.
+
+:Generator-based coroutine object:
+    Returned from a generator-based coroutine function.
+
 :Coroutine:
-    A coroutine function, or just "coroutine", is declared with ``async
+    Either *native coroutine* or *generator-based coroutine*.
-    def``. It uses ``await`` and ``return value``; see `New Coroutine
-    Declaration Syntax`_ for details.
 
 :Coroutine object:
-    Returned from a coroutine function. See `Await Expression`_ for
+    Either *native coroutine object* or *generator-based coroutine
-    details.
+    object*.
 
 :Future-like object:
     An object with an ``__await__`` method, or a C object with
@@ -662,10 +728,6 @@ Glossary
     A *Future-like* object or a *coroutine object*.  See `Await
     Expression`_ for details.
 
-:Generator-based coroutine:
-    Coroutines based in generator syntax.  Most common example is
-    ``@asyncio.coroutine``.
-
 :Asynchronous context manager:
    An asynchronous context manager has ``__aenter__`` and ``__aexit__``
    methods and can be used with ``async with``.  See `Asynchronous
@@ -696,7 +758,7 @@ generator         yield, yield from, return value     await
 
 Where:
 
-* "async def func": coroutine;
+* "async def func": native coroutine;
 
 * "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,
   ``__aexit__`` defined with the ``async`` keyword;
@@ -720,12 +782,13 @@ keywords, it was decided to modify ``tokenizer.c`` in such a way, that
 it:
 
 * recognizes ``async def`` name tokens combination (start of a
-  coroutine);
+  native coroutine);
 
-* keeps track of regular functions and coroutines;
+* keeps track of regular functions and native coroutines;
 
 * replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with
-  ``AWAIT`` when in the process of yielding tokens for coroutines.
+  ``AWAIT`` when in the process of yielding tokens for native
+  coroutines.
 
 This approach allows for seamless combination of new syntax features
 (all of them available only in ``async`` functions) with any existing
@@ -749,6 +812,34 @@ Backwards Compatibility
 This proposal preserves 100% backwards compatibility.
 
 
+asyncio
+-------
+
+``asyncio`` module was adapted and tested to work with coroutines and
+new statements.  Backwards compatibility is 100% preserved, i.e. all
+existing code will work as-is.
+
+The required changes are mainly:
+
+1. Modify ``@asyncio.coroutine`` decorator to use new
+   ``types.coroutine()`` function.
+
+2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class.
+
+3. Add ``ensure_task()`` as an alias for ``async()`` function.
+   Deprecate ``async()`` function.
+
+
+Migration strategy
+''''''''''''''''''
+
+Because *plain generators* cannot ``yield from`` *native coroutine
+objects* (see `Differences from generators`_ section for more details),
+it is advised to make sure that all generator-based coroutines are
+decorated with ``@asyncio.coroutine`` *before* starting to use the new
+syntax.
+
+
 Grammar Updates
 ---------------
 
@@ -811,23 +902,6 @@ and 3.6. In 3.7 we will transform them to proper keywords.  Making
 for people to port their code to Python 3.
 
 
-asyncio
--------
-
-``asyncio`` module was adapted and tested to work with coroutines and
-new statements.  Backwards compatibility is 100% preserved.
-
-The required changes are mainly:
-
-1. Modify ``@asyncio.coroutine`` decorator to use new
-   ``types.coroutine()`` function.
-
-2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class.
-
-3. Add ``ensure_task()`` as an alias for ``async()`` function.
-   Deprecate ``async()`` function.
-
-
 Design Considerations
 =====================
 
@@ -928,31 +1002,6 @@ in the implementation of current generator objects.  This is a matter
 for a separate PEP.
 
 
-No implicit wrapping in Futures
--------------------------------
-
-There is a proposal to add similar mechanism to ECMAScript 7 [2]_.  A
-key difference is that JavaScript "async functions" always return a
-Promise. While this approach has some advantages, it also implies that
-a new Promise object is created on each "async function" invocation.
-
-We could implement a similar functionality in Python, by wrapping all
-coroutines in a Future object, but this has the following
-disadvantages:
-
-1. Performance.  A new Future object would be instantiated on each
-   coroutine call.  Moreover, this makes implementation of ``await``
-   expressions slower (disabling optimizations of ``yield from``).
-
-2. A new built-in ``Future`` object would need to be added.
-
-3. Coming up with a generic ``Future`` interface that is usable for any
-   use case in any framework is a very hard to solve problem.
-
-4. It is not a feature that is used frequently, when most of the code
-   is coroutines.
-
-
 Why "async" and "await" keywords
 --------------------------------
 
@@ -978,8 +1027,8 @@ async/await, and because it makes working with many languages in one
 project easier (Python with ECMAScript 7 for instance).
 
 
-Why "__aiter__" is a coroutine
+Why "__aiter__" returns awaitable
-------------------------------
+---------------------------------
 
 In principle, ``__aiter__`` could be a regular function.  There are
 several good reasons to make it a coroutine:
@@ -1098,9 +1147,9 @@ This approach has the following downsides:
   returning a Future-like objects from ``__enter__`` and/or
   ``__exit__`` in Python <= 3.4;
 
-* one of the main points of this proposal is to make coroutines as
+* one of the main points of this proposal is to make native coroutines
-  simple and foolproof as possible, hence the clear separation of the
+  as simple and foolproof as possible, hence the clear separation of
-  protocols.
+  the protocols.
 
 
 Why not reuse existing "for" and "with" statements
@@ -1120,8 +1169,8 @@ Syntax for asynchronous comprehensions could be provided, but this
 construct is outside of the scope of this PEP.
 
 
-Async lambdas
+Async lambda functions
--------------
+----------------------
 
 Syntax for asynchronous lambda functions could be provided, but this
 construct is outside of the scope of this PEP.
@@ -1236,9 +1285,11 @@ List of high-level changes and new protocols
 
 6. New functions: ``sys.set_coroutine_wrapper(callback)``,
    ``sys.get_coroutine_wrapper()``, ``types.coroutine(gen)``,
-   ``inspect.iscoroutinefunction()``, and ``inspect.iscoroutine()``.
+   ``inspect.iscoroutinefunction()``, ``inspect.iscoroutine()``,
+   and ``inspect.isawaitable()``.
 
-7. New ``CO_COROUTINE`` bit flag for code objects.
+7. New ``CO_COROUTINE`` and ``CO_NATIVE_COROUTINE`` bit flags for code
+   objects.
 
 While the list of changes and new things is not short, it is important
 to understand, that most users will not use these features directly.
@@ -1305,6 +1356,9 @@ References
 
 .. [10] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF)
 
+.. [11] https://docs.python.org/3/reference/expressions.html#generator-iterator-methods
+
+.. [12] https://docs.python.org/3/reference/expressions.html#primaries
 
 Acknowledgments
 ===============