diff --git a/pep-0511.txt b/pep-0511.txt index ea422dd58..b8861448b 100644 --- a/pep-0511.txt +++ b/pep-0511.txt @@ -12,14 +12,301 @@ Python-Version: 3.6 Abstract ======== -Propose an API to support AST transformers. +Propose an API to support AST transformers. Add also ``-o OPTIM_TAG`` +command line option to change ``.pyc`` filenames. Raise an +``ImportError`` exception on import if the ``.pyc`` file is missing and +the AST transformers required to transform the code are missing. +AST transformers are not needed code transformed ahead of time (loaded +from ``.pyc`` files). -Work In Progress -================ +Rationale +========= -This document is just to reserve a PEP number, the PEP is not ready for a -public review yet. +Python does not provide a standard way to transform the code. Projects +transforming the code use various hooks. The MacroPy project uses an +import hook: it adds its own module finder in ``sys.meta_path`` to +hook its AST transformer. Another option is to monkey-patch the +builtin ``compile()`` function. There are even more options to +hook a code transformer. + +Transforming the code allows to extend the Python language for specific +use cases. Transforming an Abstract Syntax Tree (AST) is a convenient +way to implement an optimizer. It's easier to work on the AST than +working on the bytecode, AST contains more information and is more high +level. + +Python 3.6 optimizes the code using a peephole optimizer. By +definition, a peephole optimizer has a narrow view of the code and so +can only implement basic optimizations. The optimizer rewrites the +bytecode. It is difficult to enhance it, because it written in C. + +This PEP proposes to add an API to register AST transformers. + +A new ``-o OPTIM_TAG`` command line option is added to only load +transformed code: it changes the name of searched ``.pyc`` files. If the +``.pyc`` file of a module is missing and the ``.py`` is available, an +``ImportError`` exception is raised import if the AST transformers +required to transform the code are missing. The import behaviour with +the default optimizer tag (``'opt'``) is unchanged. + +The transformation can done ahead of time. It allows to implement +powerful but expensive transformations. + + +Use Cases +========= + +This section give examples of use cases explaining when and how AST +transformers will be used. + +Interactive interpreter +----------------------- + +It will be possible to use AST transformers with the interactive +interpreter which is popular in Python and commonly used to demonstrate +Python. + +The code is transformed at runtime and so the interpreter can be slower +when expensive AST transformers are used. + +Build a transformed package +--------------------------- + +It will be possible to build a package of the transformed code. + +A transformer can have a configuration. The configuration is not stored +in the package. + +All ``.pyc`` files of the package must be transformed with the same AST +transformers and the same transformers configuration. + +It is possible to build different ``.pyc`` files using different +optimizer tags. Example: ``fat`` for the default configuration and +``fat_inline`` for a different configuration with function inlining +enabled. + +A package can contain ``.pyc`` files with different optimizer tags. + + +Install a package containing transformed .pyc files +--------------------------------------------------- + +It will be possible to install a package which contains transformed +``.pyc`` files. + +All ``.pyc`` files with any optimizer tag contained in the package are +installed, not only for the current optimizer tag. + + +Build .pyc files when installing a package +------------------------------------------ + +If a package does not contain any ``.pyc`` files of the current +optimizer tag (or some ``.pyc`` files are missing), the ``.pyc`` are +created during the installation. + +AST transformers of the optimizer tag are required. Otherwise, the +installation fails with an error. + + +Execute transformed code +------------------------ + +It will be possible to execute transformed code. + +Raise an ``ImportError`` exception on import if the ``.pyc`` file of the +current optimizer tag is missing and the AST transformers required to +transform the code are missing. + +The interesting point here is that AST transformers are not needed to +execute the transformed code if all required ``.pyc`` files are already +available. + + +Changes +======= + +API to support AST transformers: + +* Add ``sys.ast_transformers``: list of AST transformers used to rewrite + an AST tree. The list is empty by default: no AST transformer. +* Add ``sys.implementation.ast_transformers``: name of AST + transformers registered in ``sys.ast_transformers`` +* Add ``sys.implementation.optim_tag`` (``str``): optimization tag. + The default optimization tag is ``'opt'``. +* Use the optimizer tag in ``.pyc`` filenames in ``importlib``. + Remove also the special case for the optimizer level ``0`` with the + default optimizer tag ``'opt'`` to simplify the code. +* Add a new ``-o OPTIM_TAG`` command line option to set + ``sys.implementation.optim_tag`` + +.. note:: + FIXME: There is a design issue: ``sys.ast_transformers`` and + ``sys.implementation.ast_transformers`` can be inconsistent. + ``sys.implementation.ast_transformers`` is required at runtime in + some corner cases to have specific code depending if a specific AST + transformer was used. Do you have a better suggestion? + +API of an AST transformer (from ``sys.ast_transformers``): + +* An AST transformer is a callable object with the prototype:: + + def ast_transformer(tree, context): + ... + return tree + + where *tree* is an AST tree and *context* is an object with a + ``filename`` attribute (``str``). New attributes may be added to + *context* in the future. + +* It must return an AST tree. +* It must have a ``name`` attribute (``str``): short string used to identify an + optimizer. The name must not contain ``.`` (dot) nor ``-`` (dash) characters: + ``.`` is used to separated fields in a ``.pyc`` filename and ``-`` is used + to join AST transformer names to build the optimizer tag. +* The transformer is called after the creation of the AST by the parser + and before the compilation to bytecode +* It can modify the AST tree in place, or create a new AST tree. + +.. note:: + It would be nice to pass the fully qualified name of a module in the + *context* when an AST transformer is used to transform a module, but + it looks like the information is not available currently. + +AST transformer changes: + +* Add a new compiler flag ``PyCF_TRANSFORMED_AST`` to get the + transformed AST. ``PyCF_ONLY_AST`` returns the AST before the + transformers. +* Add ``ast.Constant``: this type is not emited by the compiler, but + can be used in an AST transformer to simplify the code. It does not + contain line number and column offset informations on tuple or + frozenset items. +* ``PyCodeObject.co_lnotab``: line number delta becomes signed to + support moving instructions (note: need to modify MAGIC_NUMBER in + importlib). +* Enhance the bytecode compiler to support ``tuple`` and ``frozenset`` + constants. Currently, ``tuple`` and ``frozenset`` constants are + created by the peephole transformer, after the bytecode compilation. +* ``marshal`` module: fix serialization of the empty frozenset singleton +* update ``Tools/parser/unparse.py`` to support the new ``ast.Constant`` + node type + + +Example +======= + +.pyc filenames +-------------- + +Example of ``.pyc`` filenames of the ``os`` module. + +With the default optimizer tag ``'opt'``: + +=========================== ================== +.pyc filename Optimization level +=========================== ================== +``os.cpython-36.opt-0.pyc`` 0 +``os.cpython-36.opt-1.pyc`` 1 +``os.cpython-36.opt-2.pyc`` 2 +=========================== ================== + +With the ``'fat'`` optimizer tag: + +=========================== ================== +.pyc filename Optimization level +=========================== ================== +``os.cpython-36.fat-0.pyc`` 0 +``os.cpython-36.fat-1.pyc`` 1 +``os.cpython-36.fat-2.pyc`` 2 +=========================== ================== + + +AST transformer +---------------- + +Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``:: + + import ast + import sys + + + class KnightsWhoSayNi(ast.NodeTransformer): + def visit_Str(self, node): + node.s = 'Ni! Ni! Ni!' + return node + + + class ASTTransformer: + name = "knights_who_say_ni" + + def __call__(self, tree, context): + KnightsWhoSayNi().visit(tree) + return tree + + + # register the AST transformer + sys.ast_transformers.append(ASTTransformer()) + + # execute code which will be transformed by ast_transformer() + exec("print('Hello World!')") + +Output:: + + Ni! Ni! Ni! + + +Prior Art +========= + +AST optimizers +-------------- + +In 2011, Eugene Toder proposed to rewrite some peephole optimizations in +a new AST optimizer: issue #11549, `Build-out an AST optimizer, moving +some functionality out of the peephole optimizer +`_. The patch adds ``ast.Lit`` (it +was proposed to rename it to ``ast.Literal``). + +In 2012, Victor Stinner wrote the `astoptimizer +`_ project, an AST optimizer +implementing various optimizations. Most interesting optimizations break +the Python semantics since no guard is used to disable optimization if +something changes. + +Issue #17515: `Add sys.setasthook() to allow to use a custom AST +optimizer `_. + + +Python Preprocessors +-------------------- + +* `MacroPy `_: MacroPy is an + implementation of Syntactic Macros in the Python Programming Language. + MacroPy provides a mechanism for user-defined functions (macros) to + perform transformations on the abstract syntax tree (AST) of a Python + program at import time. +* `pypreprocessor `_: C-style + preprocessor directives in Python, like ``#define`` and ``#ifdef`` + + +Modify the bytecode +------------------- + +* `codetransformer `_: + Bytecode transformers for CPython inspired by the ``ast`` module’s + ``NodeTransformer``. +* `byteplay `_: Byteplay lets you + convert Python code objects into equivalent objects which are easy to + play with, and lets you convert those objects back into living Python + code objects. It's useful for applying crazy transformations on Python + functions, and is also useful in learning Python byte code + intricacies. See `byteplay documentation + `_. + +See also `BytecodeAssembler +`_. Copyright