From ac6203d00f7b5932dee40374fa484b3a3271073f Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 15 Jan 2016 01:44:58 +0100 Subject: [PATCH] PEP 511 * Fix the API: use sys.get/set_ast_transformers() * elaborate changes on importlib, try to be more concrete * add more examples of usage of AST transformers --- pep-0511.txt | 150 ++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 107 insertions(+), 43 deletions(-) diff --git a/pep-0511.txt b/pep-0511.txt index 54a945e44..762cbd8c2 100644 --- a/pep-0511.txt +++ b/pep-0511.txt @@ -37,15 +37,40 @@ importing modules, see described use cases below. Usage 1: AST optimizer ---------------------- +Python 3.6 optimizes the code using a peephole optimizer. By +definition, a peephole optimizer has a narrow view of the code and so +can only implement basic optimizations. The optimizer rewrites the +bytecode. It is difficult to enhance it, because it written in C. + Transforming an Abstract Syntax Tree (AST) is a convenient way to implement an optimizer. It's easier to work on the AST than working on the bytecode, AST contains more information and is more high level. -Python 3.6 optimizes the code using a peephole optimizer. By -definition, a peephole optimizer has a narrow view of the code and so -can only implement basic optimizations. The optimizer rewrites the -bytecode. It is difficult to enhance it, because it written in C. +Example of optimizations which can be implemented with an AST optimizer: + +* `Copy propagation + `_: + replace ``x=1; y=x`` with ``x=1; y=1`` +* `Constant folding + `_: + replace ``1+1`` with ``2`` +* `Dead code elimination + `_ + +Using guards (see the PEP 510), it is possible to implement a much wider choice +of optimizations. Examples: + +* Simplify iterable: replace ``range(3)`` with ``(0, 1, 2)`` when used + as iterable +* `Loop unrolling `_ +* Call pure builtins: replace ``len("abc")`` with ``3`` +* Copy used builtin symbols to constants + +See also `optimizations implemented in fatoptimizer +`_, a +static optimizer for Python 3.6. + Usage 2: Preprocessor --------------------- @@ -53,14 +78,17 @@ Usage 2: Preprocessor A preprocessor can be easily implemented with an AST transformer. A preprocessor has various and different usages. Examples: -* Remove debug code (like assertions and logs) to make the code faster to run +* Remove debug code like assertions and logs to make the code faster to run it for production. * `Tail-call Optimization `_ * Add profiling code -* Lazy macro create a memoizing thunk. - -Examples extending or changing the Python language: - +* `Lazy evaluation `_: + see `lazy_python `_ + (bytecode transformer) and `lazy macro of MacroPy + `_ (AST transformer) +* Change dictionary literals into collection.OrderedDict instances +* Declare constants: see `@asconstants of codetransformer + `_ * Domain Specific Language (DSL) like SQL queries. The Python language itself doesn't need to be modified. Previous attempts to implement DSL for SQL like `PEP 335 - Overloadable Boolean Operators @@ -69,7 +97,8 @@ Examples extending or changing the Python language: * String Interpolation, but `PEP 498 -- Literal String Interpolation `_ was merged into Python 3.6. -MacroPy has a much longer list of examples and use cases. +`MacroPy `_ has a long list of +examples and use cases. Use Cases @@ -147,38 +176,25 @@ Changes This PEP proposes to add an API to register AST transformers. -A new ``-o OPTIM_TAG`` command line option is added to only load -transformed code: it changes the name of searched ``.pyc`` files. If the -``.pyc`` file of a module is missing and the ``.py`` is available, an -``ImportError`` exception is raised import if the AST transformers -required to transform the code are missing. The import behaviour with -the default optimizer tag (``'opt'``) is unchanged. - The transformation can done ahead of time. It allows to implement powerful but expensive transformations. -API to support AST transformers: -* Add ``sys.ast_transformers``: list of AST transformers used to rewrite - an AST tree. The list is empty by default: no AST transformer. -* Add ``sys.implementation.ast_transformers``: name of AST - transformers registered in ``sys.ast_transformers`` -* Add ``sys.implementation.optim_tag`` (``str``): optimization tag. - The default optimization tag is ``'opt'``. -* Use the optimizer tag in ``.pyc`` filenames in ``importlib``. - Remove also the special case for the optimizer level ``0`` with the - default optimizer tag ``'opt'`` to simplify the code. -* Add a new ``-o OPTIM_TAG`` command line option to set - ``sys.implementation.optim_tag`` +API for AST transformers +------------------------ -.. note:: - FIXME: There is a design issue: ``sys.ast_transformers`` and - ``sys.implementation.ast_transformers`` can be inconsistent. - ``sys.implementation.ast_transformers`` is required at runtime in - some corner cases to have specific code depending if a specific AST - transformer was used. Do you have a better suggestion? +Add new functions to register AST transformers: -API of an AST transformer (from ``sys.ast_transformers``): +* ``sys.set_ast_transformers(transformers)``: set the list of AST + transformers +* ``sys.get_ast_transformers()``: get the list of AST + transformers. + +The order of AST transformers matter. Running transformer A and then +transformer B can give a different output than running transformer B an +then transformer A. + +API of an AST transformer: * An AST transformer is a callable object with the prototype:: @@ -202,9 +218,55 @@ API of an AST transformer (from ``sys.ast_transformers``): .. note:: It would be nice to pass the fully qualified name of a module in the *context* when an AST transformer is used to transform a module, but - it looks like the information is not available currently. + it looks like the information is not available in + ``PyParser_ASTFromStringObject()``. -AST transformer changes: + +Optimizer tag +------------- + +Changes: + +* Add ``sys.implementation.optim_tag`` (``str``): optimization tag. + The default optimization tag is ``'opt'``. +* Add a new ``-o OPTIM_TAG`` command line option to set + ``sys.implementation.optim_tag`` + +Changes on ``importlib``: + +* ``importlib`` uses ``sys.implementation.optim_tag`` to build the + ``.pyc`` filename to importing modules, instead of always using + ``opt``. Remove also the special case for the optimizer level ``0`` + with the default optimizer tag ``'opt'`` to simplify the code. +* When loading a module, if the ``.pyc`` file is missing but the ``.py`` + is available, the ``.py`` is only used if AST optimizers have the same + optimizer tag than the current tag, otherwise an ``ImportError`` + exception is raised. + +Pseudo-code of a ``use_py()`` function to decide if a ``.py`` file can +be compiled to import a module:: + + def get_ast_optim_tag(): + transformers = sys.get_ast_transformers() + if not transformers: + return 'opt' + return '-'.join(transformer.name for transformer in transformers) + + def use_py(): + return (get_ast_transformers() == sys.implementation.optim_tag) + +The order of ``sys.get_ast_transformers()`` matter. For example, the +``fat`` transformer followed by the ``pythran`` transformer gives the +optimizer tag ``fat-pythran``. + +The behaviour of the ``importlib`` module is unchanged with the default +optimizer tag (``'opt'``). + + +AST enhancements +---------------- + +Enhancements to simplify the implementation of AST transformers: * Add a new compiler flag ``PyCF_TRANSFORMED_AST`` to get the transformed AST. ``PyCF_ONLY_AST`` returns the AST before the @@ -215,9 +277,8 @@ AST transformer changes: frozenset items. * ``PyCodeObject.co_lnotab``: line number delta becomes signed to support moving instructions (note: need to modify MAGIC_NUMBER in - importlib). Implemented in the `issue #26107: code.co_lnotab: use - signed line number delta to support moving instructions in an - optimizer `_ + importlib). Implemented in the `issue #26107 + `_ * Enhance the bytecode compiler to support ``tuple`` and ``frozenset`` constants. Currently, ``tuple`` and ``frozenset`` constants are created by the peephole transformer, after the bytecode compilation. @@ -273,13 +334,16 @@ Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``:: class ASTTransformer: name = "knights_who_say_ni" + def __init__(self): + self.transformer = KnightsWhoSayNi() + def __call__(self, tree, context): - KnightsWhoSayNi().visit(tree) + self.transformer.visit(tree) return tree # register the AST transformer - sys.ast_transformers.append(ASTTransformer()) + sys.set_ast_transformers([ASTTransformer()]) # execute code which will be transformed by ast_transformer() exec("print('Hello World!')")