python-peps/pep-0511.txt

PEP: 511
Title: API for AST transformers
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 4-January-2016
Python-Version: 3.6

Abstract
========

Propose an API to support AST transformers. Add also ``-o OPTIM_TAG``
command line option to change ``.pyc`` filenames. Raise an
``ImportError`` exception on import if the ``.pyc`` file is missing and
the AST transformers required to transform the code are missing.
AST transformers are not needed code transformed ahead of time (loaded
from ``.pyc`` files).


Rationale
=========

Python does not provide a standard way to transform the code. Projects
transforming the code use various hooks. The MacroPy project uses an
import hook: it adds its own module finder in ``sys.meta_path`` to
hook its AST transformer. Another option is to monkey-patch the
builtin ``compile()`` function. There are even more options to
hook a code transformer.

Transforming the code allows to extend the Python language for specific
use cases. Transforming an Abstract Syntax Tree (AST) is a convenient
way to implement an optimizer. It's easier to work on the AST than
working on the bytecode, AST contains more information and is more high
level.

Python 3.6 optimizes the code using a peephole optimizer. By
definition, a peephole optimizer has a narrow view of the code and so
can only implement basic optimizations. The optimizer rewrites the
bytecode. It is difficult to enhance it, because it written in C.

This PEP proposes to add an API to register AST transformers.

A new ``-o OPTIM_TAG`` command line option is added to only load
transformed code: it changes the name of searched ``.pyc`` files. If the
``.pyc`` file of a module is missing and the ``.py`` is available, an
``ImportError`` exception is raised import if the AST transformers
required to transform the code are missing. The import behaviour with
the default optimizer tag (``'opt'``) is unchanged.

The transformation can done ahead of time. It allows to implement
powerful but expensive transformations.


Use Cases
=========

This section give examples of use cases explaining when and how AST
transformers will be used.

Interactive interpreter
-----------------------

It will be possible to use AST transformers with the interactive
interpreter which is popular in Python and commonly used to demonstrate
Python.

The code is transformed at runtime and so the interpreter can be slower
when expensive AST transformers are used.

Build a transformed package
---------------------------

It will be possible to build a package of the transformed code.

A transformer can have a configuration. The configuration is not stored
in the package.

All ``.pyc`` files of the package must be transformed with the same AST
transformers and the same transformers configuration.

It is possible to build different ``.pyc`` files using different
optimizer tags. Example: ``fat`` for the default configuration and
``fat_inline`` for a different configuration with function inlining
enabled.

A package can contain ``.pyc`` files with different optimizer tags.


Install a package containing transformed .pyc files
---------------------------------------------------

It will be possible to install a package which contains transformed
``.pyc`` files.

All ``.pyc`` files with any optimizer tag contained in the package are
installed, not only for the current optimizer tag.


Build .pyc files when installing a package
------------------------------------------

If a package does not contain any ``.pyc`` files of the current
optimizer tag (or some ``.pyc`` files are missing), the ``.pyc`` are
created during the installation.

AST transformers of the optimizer tag are required. Otherwise, the
installation fails with an error.


Execute transformed code
------------------------

It will be possible to execute transformed code.

Raise an ``ImportError`` exception on import if the ``.pyc`` file of the
current optimizer tag is missing and the AST transformers required to
transform the code are missing.

The interesting point here is that AST transformers are not needed to
execute the transformed code if all required ``.pyc`` files are already
available.


Changes
=======

API to support AST transformers:

* Add ``sys.ast_transformers``: list of AST transformers used to rewrite
  an AST tree. The list is empty by default: no AST transformer.
* Add ``sys.implementation.ast_transformers``: name of AST
  transformers registered in ``sys.ast_transformers``
* Add ``sys.implementation.optim_tag`` (``str``): optimization tag.
  The default optimization tag is ``'opt'``.
* Use the optimizer tag in ``.pyc`` filenames in ``importlib``.
  Remove also the special case for the optimizer level ``0`` with the
  default optimizer tag ``'opt'`` to simplify the code.
* Add a new ``-o OPTIM_TAG`` command line option to set
  ``sys.implementation.optim_tag``

.. note::
   FIXME: There is a design issue: ``sys.ast_transformers`` and
   ``sys.implementation.ast_transformers`` can be inconsistent.
   ``sys.implementation.ast_transformers`` is required at runtime in
   some corner cases to have specific code depending if a specific AST
   transformer was used. Do you have a better suggestion?

API of an AST transformer (from ``sys.ast_transformers``):

* An AST transformer is a callable object with the prototype::

    def ast_transformer(tree, context):
        ...
        return tree

  where *tree* is an AST tree and *context* is an object with a
  ``filename`` attribute (``str``). New attributes may be added to
  *context* in the future.

* It must return an AST tree.
* It must have a ``name`` attribute (``str``): short string used to identify an
  optimizer. The name must not contain ``.`` (dot) nor ``-`` (dash) characters:
  ``.`` is used to separated fields in a ``.pyc`` filename and ``-`` is used
  to join AST transformer names to build the optimizer tag.
* The transformer is called after the creation of the AST by the parser
  and before the compilation to bytecode
* It can modify the AST tree in place, or create a new AST tree.

.. note::
   It would be nice to pass the fully qualified name of a module in the
   *context*  when an AST transformer is used to transform a module, but
   it looks like the information is not available currently.

AST transformer changes:

* Add a new compiler flag ``PyCF_TRANSFORMED_AST`` to get the
  transformed AST. ``PyCF_ONLY_AST`` returns the AST before the
  transformers.
* Add ``ast.Constant``: this type is not emited by the compiler, but
  can be used in an AST transformer to simplify the code. It does not
  contain line number and column offset informations on tuple or
  frozenset items.
* ``PyCodeObject.co_lnotab``: line number delta becomes signed to
  support moving instructions (note: need to modify MAGIC_NUMBER in
  importlib). Implemented in the `issue #26107: code.co_lnotab: use
  signed line number delta to support moving instructions in an
  optimizer <https://bugs.python.org/issue26107>`_
* Enhance the bytecode compiler to support ``tuple`` and ``frozenset``
  constants. Currently, ``tuple`` and ``frozenset`` constants are
  created by the peephole transformer, after the bytecode compilation.
* ``marshal`` module: fix serialization of the empty frozenset singleton
* update ``Tools/parser/unparse.py`` to support the new ``ast.Constant``
  node type


Example
=======

.pyc filenames
--------------

Example of ``.pyc`` filenames of the ``os`` module.

With the default optimizer tag ``'opt'``:

===========================   ==================
.pyc filename                 Optimization level
===========================   ==================
``os.cpython-36.opt-0.pyc``                    0
``os.cpython-36.opt-1.pyc``                    1
``os.cpython-36.opt-2.pyc``                    2
===========================   ==================

With the ``'fat'`` optimizer tag:

===========================   ==================
.pyc filename                 Optimization level
===========================   ==================
``os.cpython-36.fat-0.pyc``                    0
``os.cpython-36.fat-1.pyc``                    1
``os.cpython-36.fat-2.pyc``                    2
===========================   ==================


AST transformer
----------------

Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``::

    import ast
    import sys


    class KnightsWhoSayNi(ast.NodeTransformer):
        def visit_Str(self, node):
            node.s = 'Ni! Ni! Ni!'
            return node


    class ASTTransformer:
        name = "knights_who_say_ni"

        def __call__(self, tree, context):
            KnightsWhoSayNi().visit(tree)
            return tree


    # register the AST transformer
    sys.ast_transformers.append(ASTTransformer())

    # execute code which will be transformed by ast_transformer()
    exec("print('Hello World!')")

Output::

    Ni! Ni! Ni!


Prior Art
=========

AST optimizers
--------------

In 2011, Eugene Toder proposed to rewrite some peephole optimizations in
a new AST optimizer: issue #11549, `Build-out an AST optimizer, moving
some functionality out of the peephole optimizer
<https://bugs.python.org/issue11549>`_.  The patch adds ``ast.Lit`` (it
was proposed to rename it to ``ast.Literal``).

In 2012, Victor Stinner wrote the `astoptimizer
<https://bitbucket.org/haypo/astoptimizer/>`_ project, an AST optimizer
implementing various optimizations. Most interesting optimizations break
the Python semantics since no guard is used to disable optimization if
something changes.

Issue #17515: `Add sys.setasthook() to allow to use a custom AST
optimizer <https://bugs.python.org/issue17515>`_.


Python Preprocessors
--------------------

* `MacroPy <https://github.com/lihaoyi/macropy>`_: MacroPy is an
  implementation of Syntactic Macros in the Python Programming Language.
  MacroPy provides a mechanism for user-defined functions (macros) to
  perform transformations on the abstract syntax tree (AST) of a Python
  program at import time.
* `pypreprocessor <https://code.google.com/p/pypreprocessor/>`_: C-style
  preprocessor directives in Python, like ``#define`` and ``#ifdef``


Modify the bytecode
-------------------

* `codetransformer <https://pypi.python.org/pypi/codetransformer>`_:
  Bytecode transformers for CPython inspired by the ``ast`` module’s
  ``NodeTransformer``.
* `byteplay <http://code.google.com/p/byteplay/>`_: Byteplay lets you
  convert Python code objects into equivalent objects which are easy to
  play with, and lets you convert those objects back into living Python
  code objects. It's useful for applying crazy transformations on Python
  functions, and is also useful in learning Python byte code
  intricacies. See `byteplay documentation
  <http://wiki.python.org/moin/ByteplayDoc>`_.

See also `BytecodeAssembler
<http://pypi.python.org/pypi/BytecodeAssembler>`_.


Copyright
=========

This document has been placed in the public domain.