316 lines
11 KiB
Plaintext
316 lines
11 KiB
Plaintext
PEP: 511
|
||
Title: API for AST transformers
|
||
Version: $Revision$
|
||
Last-Modified: $Date$
|
||
Author: Victor Stinner <victor.stinner@gmail.com>
|
||
Status: Draft
|
||
Type: Standards Track
|
||
Content-Type: text/x-rst
|
||
Created: 4-January-2016
|
||
Python-Version: 3.6
|
||
|
||
Abstract
|
||
========
|
||
|
||
Propose an API to support AST transformers. Add also ``-o OPTIM_TAG``
|
||
command line option to change ``.pyc`` filenames. Raise an
|
||
``ImportError`` exception on import if the ``.pyc`` file is missing and
|
||
the AST transformers required to transform the code are missing.
|
||
AST transformers are not needed code transformed ahead of time (loaded
|
||
from ``.pyc`` files).
|
||
|
||
|
||
Rationale
|
||
=========
|
||
|
||
Python does not provide a standard way to transform the code. Projects
|
||
transforming the code use various hooks. The MacroPy project uses an
|
||
import hook: it adds its own module finder in ``sys.meta_path`` to
|
||
hook its AST transformer. Another option is to monkey-patch the
|
||
builtin ``compile()`` function. There are even more options to
|
||
hook a code transformer.
|
||
|
||
Transforming the code allows to extend the Python language for specific
|
||
use cases. Transforming an Abstract Syntax Tree (AST) is a convenient
|
||
way to implement an optimizer. It's easier to work on the AST than
|
||
working on the bytecode, AST contains more information and is more high
|
||
level.
|
||
|
||
Python 3.6 optimizes the code using a peephole optimizer. By
|
||
definition, a peephole optimizer has a narrow view of the code and so
|
||
can only implement basic optimizations. The optimizer rewrites the
|
||
bytecode. It is difficult to enhance it, because it written in C.
|
||
|
||
This PEP proposes to add an API to register AST transformers.
|
||
|
||
A new ``-o OPTIM_TAG`` command line option is added to only load
|
||
transformed code: it changes the name of searched ``.pyc`` files. If the
|
||
``.pyc`` file of a module is missing and the ``.py`` is available, an
|
||
``ImportError`` exception is raised import if the AST transformers
|
||
required to transform the code are missing. The import behaviour with
|
||
the default optimizer tag (``'opt'``) is unchanged.
|
||
|
||
The transformation can done ahead of time. It allows to implement
|
||
powerful but expensive transformations.
|
||
|
||
|
||
Use Cases
|
||
=========
|
||
|
||
This section give examples of use cases explaining when and how AST
|
||
transformers will be used.
|
||
|
||
Interactive interpreter
|
||
-----------------------
|
||
|
||
It will be possible to use AST transformers with the interactive
|
||
interpreter which is popular in Python and commonly used to demonstrate
|
||
Python.
|
||
|
||
The code is transformed at runtime and so the interpreter can be slower
|
||
when expensive AST transformers are used.
|
||
|
||
Build a transformed package
|
||
---------------------------
|
||
|
||
It will be possible to build a package of the transformed code.
|
||
|
||
A transformer can have a configuration. The configuration is not stored
|
||
in the package.
|
||
|
||
All ``.pyc`` files of the package must be transformed with the same AST
|
||
transformers and the same transformers configuration.
|
||
|
||
It is possible to build different ``.pyc`` files using different
|
||
optimizer tags. Example: ``fat`` for the default configuration and
|
||
``fat_inline`` for a different configuration with function inlining
|
||
enabled.
|
||
|
||
A package can contain ``.pyc`` files with different optimizer tags.
|
||
|
||
|
||
Install a package containing transformed .pyc files
|
||
---------------------------------------------------
|
||
|
||
It will be possible to install a package which contains transformed
|
||
``.pyc`` files.
|
||
|
||
All ``.pyc`` files with any optimizer tag contained in the package are
|
||
installed, not only for the current optimizer tag.
|
||
|
||
|
||
Build .pyc files when installing a package
|
||
------------------------------------------
|
||
|
||
If a package does not contain any ``.pyc`` files of the current
|
||
optimizer tag (or some ``.pyc`` files are missing), the ``.pyc`` are
|
||
created during the installation.
|
||
|
||
AST transformers of the optimizer tag are required. Otherwise, the
|
||
installation fails with an error.
|
||
|
||
|
||
Execute transformed code
|
||
------------------------
|
||
|
||
It will be possible to execute transformed code.
|
||
|
||
Raise an ``ImportError`` exception on import if the ``.pyc`` file of the
|
||
current optimizer tag is missing and the AST transformers required to
|
||
transform the code are missing.
|
||
|
||
The interesting point here is that AST transformers are not needed to
|
||
execute the transformed code if all required ``.pyc`` files are already
|
||
available.
|
||
|
||
|
||
Changes
|
||
=======
|
||
|
||
API to support AST transformers:
|
||
|
||
* Add ``sys.ast_transformers``: list of AST transformers used to rewrite
|
||
an AST tree. The list is empty by default: no AST transformer.
|
||
* Add ``sys.implementation.ast_transformers``: name of AST
|
||
transformers registered in ``sys.ast_transformers``
|
||
* Add ``sys.implementation.optim_tag`` (``str``): optimization tag.
|
||
The default optimization tag is ``'opt'``.
|
||
* Use the optimizer tag in ``.pyc`` filenames in ``importlib``.
|
||
Remove also the special case for the optimizer level ``0`` with the
|
||
default optimizer tag ``'opt'`` to simplify the code.
|
||
* Add a new ``-o OPTIM_TAG`` command line option to set
|
||
``sys.implementation.optim_tag``
|
||
|
||
.. note::
|
||
FIXME: There is a design issue: ``sys.ast_transformers`` and
|
||
``sys.implementation.ast_transformers`` can be inconsistent.
|
||
``sys.implementation.ast_transformers`` is required at runtime in
|
||
some corner cases to have specific code depending if a specific AST
|
||
transformer was used. Do you have a better suggestion?
|
||
|
||
API of an AST transformer (from ``sys.ast_transformers``):
|
||
|
||
* An AST transformer is a callable object with the prototype::
|
||
|
||
def ast_transformer(tree, context):
|
||
...
|
||
return tree
|
||
|
||
where *tree* is an AST tree and *context* is an object with a
|
||
``filename`` attribute (``str``). New attributes may be added to
|
||
*context* in the future.
|
||
|
||
* It must return an AST tree.
|
||
* It must have a ``name`` attribute (``str``): short string used to identify an
|
||
optimizer. The name must not contain ``.`` (dot) nor ``-`` (dash) characters:
|
||
``.`` is used to separated fields in a ``.pyc`` filename and ``-`` is used
|
||
to join AST transformer names to build the optimizer tag.
|
||
* The transformer is called after the creation of the AST by the parser
|
||
and before the compilation to bytecode
|
||
* It can modify the AST tree in place, or create a new AST tree.
|
||
|
||
.. note::
|
||
It would be nice to pass the fully qualified name of a module in the
|
||
*context* when an AST transformer is used to transform a module, but
|
||
it looks like the information is not available currently.
|
||
|
||
AST transformer changes:
|
||
|
||
* Add a new compiler flag ``PyCF_TRANSFORMED_AST`` to get the
|
||
transformed AST. ``PyCF_ONLY_AST`` returns the AST before the
|
||
transformers.
|
||
* Add ``ast.Constant``: this type is not emited by the compiler, but
|
||
can be used in an AST transformer to simplify the code. It does not
|
||
contain line number and column offset informations on tuple or
|
||
frozenset items.
|
||
* ``PyCodeObject.co_lnotab``: line number delta becomes signed to
|
||
support moving instructions (note: need to modify MAGIC_NUMBER in
|
||
importlib).
|
||
* Enhance the bytecode compiler to support ``tuple`` and ``frozenset``
|
||
constants. Currently, ``tuple`` and ``frozenset`` constants are
|
||
created by the peephole transformer, after the bytecode compilation.
|
||
* ``marshal`` module: fix serialization of the empty frozenset singleton
|
||
* update ``Tools/parser/unparse.py`` to support the new ``ast.Constant``
|
||
node type
|
||
|
||
|
||
Example
|
||
=======
|
||
|
||
.pyc filenames
|
||
--------------
|
||
|
||
Example of ``.pyc`` filenames of the ``os`` module.
|
||
|
||
With the default optimizer tag ``'opt'``:
|
||
|
||
=========================== ==================
|
||
.pyc filename Optimization level
|
||
=========================== ==================
|
||
``os.cpython-36.opt-0.pyc`` 0
|
||
``os.cpython-36.opt-1.pyc`` 1
|
||
``os.cpython-36.opt-2.pyc`` 2
|
||
=========================== ==================
|
||
|
||
With the ``'fat'`` optimizer tag:
|
||
|
||
=========================== ==================
|
||
.pyc filename Optimization level
|
||
=========================== ==================
|
||
``os.cpython-36.fat-0.pyc`` 0
|
||
``os.cpython-36.fat-1.pyc`` 1
|
||
``os.cpython-36.fat-2.pyc`` 2
|
||
=========================== ==================
|
||
|
||
|
||
AST transformer
|
||
----------------
|
||
|
||
Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``::
|
||
|
||
import ast
|
||
import sys
|
||
|
||
|
||
class KnightsWhoSayNi(ast.NodeTransformer):
|
||
def visit_Str(self, node):
|
||
node.s = 'Ni! Ni! Ni!'
|
||
return node
|
||
|
||
|
||
class ASTTransformer:
|
||
name = "knights_who_say_ni"
|
||
|
||
def __call__(self, tree, context):
|
||
KnightsWhoSayNi().visit(tree)
|
||
return tree
|
||
|
||
|
||
# register the AST transformer
|
||
sys.ast_transformers.append(ASTTransformer())
|
||
|
||
# execute code which will be transformed by ast_transformer()
|
||
exec("print('Hello World!')")
|
||
|
||
Output::
|
||
|
||
Ni! Ni! Ni!
|
||
|
||
|
||
Prior Art
|
||
=========
|
||
|
||
AST optimizers
|
||
--------------
|
||
|
||
In 2011, Eugene Toder proposed to rewrite some peephole optimizations in
|
||
a new AST optimizer: issue #11549, `Build-out an AST optimizer, moving
|
||
some functionality out of the peephole optimizer
|
||
<https://bugs.python.org/issue11549>`_. The patch adds ``ast.Lit`` (it
|
||
was proposed to rename it to ``ast.Literal``).
|
||
|
||
In 2012, Victor Stinner wrote the `astoptimizer
|
||
<https://bitbucket.org/haypo/astoptimizer/>`_ project, an AST optimizer
|
||
implementing various optimizations. Most interesting optimizations break
|
||
the Python semantics since no guard is used to disable optimization if
|
||
something changes.
|
||
|
||
Issue #17515: `Add sys.setasthook() to allow to use a custom AST
|
||
optimizer <https://bugs.python.org/issue17515>`_.
|
||
|
||
|
||
Python Preprocessors
|
||
--------------------
|
||
|
||
* `MacroPy <https://github.com/lihaoyi/macropy>`_: MacroPy is an
|
||
implementation of Syntactic Macros in the Python Programming Language.
|
||
MacroPy provides a mechanism for user-defined functions (macros) to
|
||
perform transformations on the abstract syntax tree (AST) of a Python
|
||
program at import time.
|
||
* `pypreprocessor <https://code.google.com/p/pypreprocessor/>`_: C-style
|
||
preprocessor directives in Python, like ``#define`` and ``#ifdef``
|
||
|
||
|
||
Modify the bytecode
|
||
-------------------
|
||
|
||
* `codetransformer <https://pypi.python.org/pypi/codetransformer>`_:
|
||
Bytecode transformers for CPython inspired by the ``ast`` module’s
|
||
``NodeTransformer``.
|
||
* `byteplay <http://code.google.com/p/byteplay/>`_: Byteplay lets you
|
||
convert Python code objects into equivalent objects which are easy to
|
||
play with, and lets you convert those objects back into living Python
|
||
code objects. It's useful for applying crazy transformations on Python
|
||
functions, and is also useful in learning Python byte code
|
||
intricacies. See `byteplay documentation
|
||
<http://wiki.python.org/moin/ByteplayDoc>`_.
|
||
|
||
See also `BytecodeAssembler
|
||
<http://pypi.python.org/pypi/BytecodeAssembler>`_.
|
||
|
||
|
||
Copyright
|
||
=========
|
||
|
||
This document has been placed in the public domain.
|