iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

pep-0492: Reword tokenization algorithm description.

This commit is contained in:
Yury Selivanov 2015-04-29 23:10:23 -04:00
parent c28890fb42
commit 08ceef7eeb
1 changed files with 9 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
pep-0492.txt
View File

@ -781,14 +781,16 @@ To avoid backwards compatibility issues with ``async`` and ``await``
keywords, it was decided to modify ``tokenizer.c`` in such a way, that
it:
* recognizes ``async def`` name tokens combination (start of a
native coroutine);
* recognizes ``async def`` ``NAME`` tokens combination;
* keeps track of regular functions and native coroutines;
* keeps track of regular ``def`` and ``async def`` indented blocks;
* replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with
``AWAIT`` when in the process of yielding tokens for native
coroutines.
* while tokenizing ``async def`` block, it replaces ``'async'``
``NAME`` token with ``ASYNC``, and ``'await'`` ``NAME`` token with
``AWAIT``;
* while tokenizing ``def`` block, it yields ``'async'`` and ``'await'``
``NAME`` tokens as is.
This approach allows for seamless combination of new syntax features
(all of them available only in ``async`` functions) with any existing
@ -843,7 +845,7 @@ syntax.
Grammar Updates
---------------
Grammar changes are also fairly minimal::
Grammar changes are fairly minimal::
decorated: decorators (classdef | funcdef | async_funcdef)
async_funcdef: ASYNC funcdef