iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

pep-0492: v4.

This commit is contained in:
Yury Selivanov 2015-04-29 21:11:53 -04:00
parent ef2437d422
commit c28890fb42
1 changed files with 201 additions and 147 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

338
pep-0492.txt
View File

@ -8,7 +8,7 @@ Type: Standards Track
Content-Type: text/x-rst
Created: 09-Apr-2015
Python-Version: 3.5
Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015
Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015
Abstract
@ -57,8 +57,7 @@ Specification
=============
This proposal introduces new syntax and semantics to enhance coroutine
support in Python, it does not change the internal implementation of
coroutines, which are still based on generators.
support in Python.
This specification presumes knowledge of the implementation of
coroutines in Python (PEP 342 and PEP 380). Motivation for the syntax
@ -66,21 +65,22 @@ changes proposed here comes from the asyncio framework (PEP 3156) and
the "Cofunctions" proposal (PEP 3152, now rejected in favor of this
specification).
From this point in this document we use the word *coroutine* to refer
to functions declared using the new syntax. *generator-based
From this point in this document we use the word *native coroutine* to
refer to functions declared using the new syntax. *generator-based
coroutine* is used where necessary to refer to coroutines that are
based on generator syntax.
based on generator syntax. *coroutine* is used in contexts where both
definitions are applicable.
New Coroutine Declaration Syntax
--------------------------------
The following new syntax is used to declare a coroutine::
The following new syntax is used to declare a *native coroutine*::
async def read_data(db):
pass
Key properties of coroutines:
Key properties of *native coroutines*:
* ``async def`` functions are always coroutines, even if they do not
contain ``await`` expressions.
@ -88,10 +88,16 @@ Key properties of coroutines:
* It is a ``SyntaxError`` to have ``yield`` or ``yield from``
expressions in an ``async`` function.
* Internally, a new code object flag - ``CO_COROUTINE`` - is introduced
to enable runtime detection of coroutines (and migrating existing
code). All coroutines have both ``CO_COROUTINE`` and ``CO_GENERATOR``
flags set.
* Internally, two new code object flags were introduced:
- ``CO_COROUTINE`` is used to enable runtime detection of
*coroutines* (and migrating existing code).
- ``CO_NATIVE_COROUTINE`` is used to mark *native coroutines*
(defined with new syntax.)
All coroutines have ``CO_COROUTINE``, ``CO_NATIVE_COROUTINE``, and
``CO_GENERATOR`` flags set.
* Regular generators, when called, return a *generator object*;
similarly, coroutines return a *coroutine object*.
@ -100,15 +106,25 @@ Key properties of coroutines:
and are replaced with a ``RuntimeError``. For regular generators
such behavior requires a future import (see PEP 479).
* See also `Coroutine objects`_ section.
types.coroutine()
-----------------
A new function ``coroutine(gen)`` is added to the ``types`` module. It
applies ``CO_COROUTINE`` flag to the passed generator-function's code
object, making it to return a *coroutine object* when called.
allows interoperability between existing *generator-based coroutines*
in asyncio and *native coroutines* introduced by this PEP.
This feature enables an easy upgrade path for existing libraries.
The function applies ``CO_COROUTINE`` flag to generator-function's code
object, making it return a *coroutine object*.
The function can be used as a decorator, since it modifies generator-
functions in-place and returns them.
Note, that the ``CO_NATIVE_COROUTINE`` flag is not applied by
``types.coroutine()`` to make it possible to separate *native
coroutines* defined with new syntax, from *generator-based coroutines*.
Await Expression
@ -129,7 +145,9 @@ It uses the ``yield from`` implementation with an extra step of
validating its argument. ``await`` only accepts an *awaitable*, which
can be one of:
* A *coroutine object* returned from a *coroutine* or a generator
* A *native coroutine object* returned from a *native coroutine*.
* A *generator-based coroutine object* returned from a generator
decorated with ``types.coroutine()``.
* An object with an ``__await__`` method returning an iterator.
@ -142,7 +160,7 @@ can be one of:
explanation.)
To enable this behavior for coroutines, a new magic method called
``__await__`` is added. In asyncio, for instance, to enable Future
``__await__`` is added. In asyncio, for instance, to enable *Future*
objects in ``await`` statements, the only change is to add
``__await__ = __iter__`` line to ``asyncio.Future`` class.
@ -160,7 +178,9 @@ can be one of:
* Objects defined with CPython C API with a ``tp_await`` function,
returning an iterator (similar to ``__await__`` method).
It is a ``SyntaxError`` to use ``await`` outside of a coroutine.
It is a ``SyntaxError`` to use ``await`` outside of an ``async def``
function (like it is a ``SyntaxError`` to use ``yield`` outside of
``def`` function.)
It is a ``TypeError`` to pass anything other than an *awaitable* object
to an ``await`` expression.
@ -169,11 +189,22 @@ to an ``await`` expression.
Updated operator precedence table
'''''''''''''''''''''''''''''''''
``await`` keyword is defined differently from ``yield`` and ``yield
from`` in the Grammar.
``await`` keyword is defined as follows::
The key difference is that *await expressions* do not require
parentheses around them most of the times.
power ::= await ["**" u_expr]
await ::= ["await"] primary
where "primary" represents the most tightly bound operations of the
language. Its syntax is::
primary ::= atom | attributeref | subscription | slicing | call
See Python Documentation [12]_ and `Grammar Updates`_ section of this
proposal for details.
The key ``await`` difference from ``yield`` and ``yield from``
operators is that *await expressions* do not require parentheses around
them most of the times.
Also, ``yield from`` allows any expression as its argument, including
expressions like ``yield from a() + b()``, that would be parsed as
@ -186,7 +217,8 @@ operators.
+------------------------------+-----------------------------------+
| Operator | Description |
+==============================+===================================+
| ``yield``, ``yield from`` | Yield expression |
| ``yield`` ``x``, | Yield expression |
| ``yield from`` ``x`` | |
+------------------------------+-----------------------------------+
| ``lambda`` | Lambda expression |
+------------------------------+-----------------------------------+
@ -221,7 +253,7 @@ operators.
+------------------------------+-----------------------------------+
| ``**`` | Exponentiation |
+------------------------------+-----------------------------------+
| ``await`` | Await expression |
| ``await`` ``x`` | Await expression |
+------------------------------+-----------------------------------+
| ``x[index]``, | Subscription, slicing, |
| ``x[index:index]``, | call, attribute reference |
@ -234,8 +266,6 @@ operators.
| ``{expressions...}`` | set display |
+------------------------------+-----------------------------------+
See `Grammar Updates`_ section for details.
Examples of "await" expressions
'''''''''''''''''''''''''''''''
@ -266,8 +296,6 @@ Expression Should be written as
``await -coro()`` ``await (-coro())``
================================== ==================================
See `Grammar Updates`_ section for details.
Asynchronous Context Managers and "async with"
----------------------------------------------
@ -305,18 +333,13 @@ which is semantically equivalent to::
aenter = type(mgr).__aenter__(mgr)
exc = True
try:
try:
VAR = await aenter
BLOCK
except:
exc = False
exit_res = await aexit(mgr, *sys.exc_info())
if not exit_res:
if not await aexit(mgr, *sys.exc_info()):
raise
finally:
if exc:
else:
await aexit(mgr, None, None, None)
@ -325,13 +348,13 @@ context managers in a single ``async with`` statement.
It is an error to pass a regular context manager without ``__aenter__``
and ``__aexit__`` methods to ``async with``. It is a ``SyntaxError``
to use ``async with`` outside of a coroutine.
to use ``async with`` outside of an ``async def`` function.
Example
'''''''
With asynchronous context managers it is easy to implement proper
With *asynchronous context managers* it is easy to implement proper
database transaction managers for coroutines::
async def commit(session, data):
@ -416,7 +439,7 @@ which is semantically equivalent to::
It is a ``TypeError`` to pass a regular iterable without ``__aiter__``
method to ``async for``. It is a ``SyntaxError`` to use ``async for``
outside of a coroutine.
outside of an ``async def`` function.
As for with regular ``for`` statement, ``async for`` has an optional
``else`` clause.
@ -536,11 +559,61 @@ Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions
raised in coroutines are wrapped in ``RuntimeError``.
Coroutine objects
-----------------
Differences from generators
'''''''''''''''''''''''''''
This section applies only to *native coroutines* with
``CO_NATIVE_COROUTINE`` flag, i.e. defined with the new ``async def``
syntax.
**The behavior of existing *generator-based coroutines* in asyncio
remains unchanged.**
Great effort has been made to make sure that coroutines and
generators are treated as distinct concepts:
1. *Native coroutine objects* do not implement ``__iter__`` and
``__next__`` methods. Therefore, they cannot be iterated over or
passed to ``iter()``, ``list()``, ``tuple()`` and other built-ins.
They also cannot be used in a ``for..in`` loop.
An attempt to use ``__iter__`` or ``__next__`` on a *native
coroutine object* will result in a ``TypeError``.
2. *Plain generators* cannot ``yield from`` *native coroutine objects*:
doing so will result in a ``TypeError``.
3. *generator-based coroutines* (for asyncio code must be decorated
with ``@asyncio.coroutine``) can ``yield from`` *native coroutine
objects*.
4. ``inspect.isgenerator()`` and ``inspect.isgeneratorfunction()``
return ``False`` for *native coroutine objects* and *native
coroutine functions*.
Coroutine object methods
''''''''''''''''''''''''
Coroutines are based on generators internally, thus they share the
implementation. Similarly to generator objects, coroutine objects have
``throw()``, ``send()`` and ``close()`` methods. ``StopIteration`` and
``GeneratorExit`` play the same role for coroutine objects (although
PEP 479 is enabled by default for coroutines). See PEP 342, PEP 380,
and Python Documentation [11]_ for details.
``throw()``, ``send()`` methods for coroutine objects are used to push
values and raise errors into *Future-like* objects.
Debugging Features
------------------
One of the most frequent mistakes that people make when using
generators as coroutines is forgetting to use ``yield from``::
A common beginner mistake is forgetting to use ``yield from`` on
coroutines::
@asyncio.coroutine
def useful():
@ -590,65 +663,58 @@ Example::
# previously set wrapper
assert not isinstance(debug_me(), asyncio.CoroWrapper)
If ``sys.set_coroutine_wrapper()`` is called twice, the new wrapper
replaces the previous wrapper. ``sys.set_coroutine_wrapper(None)``
unsets the wrapper.
New Standard Library Functions
------------------------------
inspect.iscoroutine() and inspect.iscoroutineobject()
-----------------------------------------------------
Two new functions are added to the ``inspect`` module:
* ``types.coroutine(gen)``. See `types.coroutine()`_ section for
details.
* ``inspect.iscoroutine(obj)`` returns ``True`` if ``obj`` is a
coroutine object.
*coroutine object*.
* ``inspect.iscoroutinefunction(obj)`` returns ``True`` is ``obj`` is a
coroutine function.
* ``inspect.iscoroutinefunction(obj)`` returns ``True`` if ``obj`` is a
*coroutine function*.
* ``inspect.isawaitable(obj)`` returns ``True`` if ``obj`` can be used
in ``await`` expression. See `Await Expression`_ for details.
Differences between coroutines and generators
---------------------------------------------
* ``sys.set_coroutine_wrapper(wraper)`` allows to intercept creation of
*coroutine objects*. ``wraper`` must be a callable that accepts one
argument: a *coroutine object* or ``None``. ``None`` resets the
wrapper. If called twice, the new wrapper replaces the previous one.
See `Debugging Features`_ for more details.
A great effort has been made to make sure that coroutines and
generators are separate concepts:
1. Coroutine objects do not implement ``__iter__`` and ``__next__``
methods. Therefore they cannot be iterated over or passed to
``iter()``, ``list()``, ``tuple()`` and other built-ins. They
also cannot be used in a ``for..in`` loop.
2. ``yield from`` does not accept coroutine objects (unless it is used
in a generator-based coroutine decorated with ``types.coroutine``.)
3. ``yield from`` does not accept coroutine objects from plain Python
generators (*not* generator-based coroutines.)
4. ``inspect.isgenerator()`` and ``inspect.isgeneratorfunction()``
return ``False`` for coroutine objects and coroutine functions.
Coroutine objects
-----------------
Coroutines are based on generators internally, thus they share the
implementation. Similarly to generator objects, coroutine objects have
``throw``, ``send`` and ``close`` methods. ``StopIteration`` and
``GeneratorExit`` play the same role for coroutine objects (although
PEP 479 is enabled by default for coroutines).
* ``sys.get_coroutine_wrapper()`` returns the current wrapper object.
Returns ``None`` if no wrapper was set. See `Debugging Features`_
for more details.
Glossary
========
:Native coroutine:
A coroutine function is declared with ``async def``. It uses
``await`` and ``return value``; see `New Coroutine Declaration
Syntax`_ for details.
:Native coroutine object:
Returned from a native coroutine function. See `Await Expression`_
for details.
:Generator-based coroutine:
Coroutines based on generator syntax. Most common example are
functions decorated with ``@asyncio.coroutine``.
:Generator-based coroutine object:
Returned from a generator-based coroutine function.
:Coroutine:
A coroutine function, or just "coroutine", is declared with ``async
def``. It uses ``await`` and ``return value``; see `New Coroutine
Declaration Syntax`_ for details.
Either *native coroutine* or *generator-based coroutine*.
:Coroutine object:
Returned from a coroutine function. See `Await Expression`_ for
details.
Either *native coroutine object* or *generator-based coroutine
object*.
:Future-like object:
An object with an ``__await__`` method, or a C object with
@ -662,10 +728,6 @@ Glossary
A *Future-like* object or a *coroutine object*. See `Await
Expression`_ for details.
:Generator-based coroutine:
Coroutines based in generator syntax. Most common example is
``@asyncio.coroutine``.
:Asynchronous context manager:
An asynchronous context manager has ``__aenter__`` and ``__aexit__``
methods and can be used with ``async with``. See `Asynchronous
@ -696,7 +758,7 @@ generator yield, yield from, return value await
Where:
* "async def func": coroutine;
* "async def func": native coroutine;
* "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,
``__aexit__`` defined with the ``async`` keyword;
@ -720,12 +782,13 @@ keywords, it was decided to modify ``tokenizer.c`` in such a way, that
it:
* recognizes ``async def`` name tokens combination (start of a
coroutine);
native coroutine);
* keeps track of regular functions and coroutines;
* keeps track of regular functions and native coroutines;
* replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with
``AWAIT`` when in the process of yielding tokens for coroutines.
``AWAIT`` when in the process of yielding tokens for native
coroutines.
This approach allows for seamless combination of new syntax features
(all of them available only in ``async`` functions) with any existing
@ -749,6 +812,34 @@ Backwards Compatibility
This proposal preserves 100% backwards compatibility.
asyncio
-------
``asyncio`` module was adapted and tested to work with coroutines and
new statements. Backwards compatibility is 100% preserved, i.e. all
existing code will work as-is.
The required changes are mainly:
1. Modify ``@asyncio.coroutine`` decorator to use new
``types.coroutine()`` function.
2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class.
3. Add ``ensure_task()`` as an alias for ``async()`` function.
Deprecate ``async()`` function.
Migration strategy
''''''''''''''''''
Because *plain generators* cannot ``yield from`` *native coroutine
objects* (see `Differences from generators`_ section for more details),
it is advised to make sure that all generator-based coroutines are
decorated with ``@asyncio.coroutine`` *before* starting to use the new
syntax.
Grammar Updates
---------------
@ -811,23 +902,6 @@ and 3.6. In 3.7 we will transform them to proper keywords. Making
for people to port their code to Python 3.
asyncio
-------
``asyncio`` module was adapted and tested to work with coroutines and
new statements. Backwards compatibility is 100% preserved.
The required changes are mainly:
1. Modify ``@asyncio.coroutine`` decorator to use new
``types.coroutine()`` function.
2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class.
3. Add ``ensure_task()`` as an alias for ``async()`` function.
Deprecate ``async()`` function.
Design Considerations
=====================
@ -928,31 +1002,6 @@ in the implementation of current generator objects. This is a matter
for a separate PEP.
No implicit wrapping in Futures
-------------------------------
There is a proposal to add similar mechanism to ECMAScript 7 [2]_. A
key difference is that JavaScript "async functions" always return a
Promise. While this approach has some advantages, it also implies that
a new Promise object is created on each "async function" invocation.
We could implement a similar functionality in Python, by wrapping all
coroutines in a Future object, but this has the following
disadvantages:
1. Performance. A new Future object would be instantiated on each
coroutine call. Moreover, this makes implementation of ``await``
expressions slower (disabling optimizations of ``yield from``).
2. A new built-in ``Future`` object would need to be added.
3. Coming up with a generic ``Future`` interface that is usable for any
use case in any framework is a very hard to solve problem.
4. It is not a feature that is used frequently, when most of the code
is coroutines.
Why "async" and "await" keywords
--------------------------------
@ -978,8 +1027,8 @@ async/await, and because it makes working with many languages in one
project easier (Python with ECMAScript 7 for instance).
Why "__aiter__" is a coroutine
------------------------------
Why "__aiter__" returns awaitable
---------------------------------
In principle, ``__aiter__`` could be a regular function. There are
several good reasons to make it a coroutine:
@ -1098,9 +1147,9 @@ This approach has the following downsides:
returning a Future-like objects from ``__enter__`` and/or
``__exit__`` in Python <= 3.4;
* one of the main points of this proposal is to make coroutines as
simple and foolproof as possible, hence the clear separation of the
protocols.
* one of the main points of this proposal is to make native coroutines
as simple and foolproof as possible, hence the clear separation of
the protocols.
Why not reuse existing "for" and "with" statements
@ -1120,8 +1169,8 @@ Syntax for asynchronous comprehensions could be provided, but this
construct is outside of the scope of this PEP.
Async lambdas
-------------
Async lambda functions
----------------------
Syntax for asynchronous lambda functions could be provided, but this
construct is outside of the scope of this PEP.
@ -1236,9 +1285,11 @@ List of high-level changes and new protocols
6. New functions: ``sys.set_coroutine_wrapper(callback)``,
``sys.get_coroutine_wrapper()``, ``types.coroutine(gen)``,
``inspect.iscoroutinefunction()``, and ``inspect.iscoroutine()``.
``inspect.iscoroutinefunction()``, ``inspect.iscoroutine()``,
and ``inspect.isawaitable()``.
7. New ``CO_COROUTINE`` bit flag for code objects.
7. New ``CO_COROUTINE`` and ``CO_NATIVE_COROUTINE`` bit flags for code
objects.
While the list of changes and new things is not short, it is important
to understand, that most users will not use these features directly.
@ -1305,6 +1356,9 @@ References
.. [10] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF)
.. [11] https://docs.python.org/3/reference/expressions.html#generator-iterator-methods
.. [12] https://docs.python.org/3/reference/expressions.html#primaries
Acknowledgments
===============