PEP 511: add bytecode transformers

pep-0511.txt

@@ -1,5 +1,5 @@
 PEP: 511
-Title: API for AST transformers
+Title: API for code transformers
 Version: $Revision$
 Last-Modified: $Date$
 Author: Victor Stinner <victor.stinner@gmail.com>
@@ -12,12 +12,12 @@ Python-Version: 3.6
 Abstract
 ========
 
-Propose an API to support AST transformers. Add also ``-o OPTIM_TAG``
+Propose an API to register AST and bytecode transformers. Add also ``-o
-command line option to change ``.pyc`` filenames. Raise an
+OPTIM_TAG`` command line option to change ``.pyc`` filenames. Raise an
 ``ImportError`` exception on import if the ``.pyc`` file is missing and
-the AST transformers required to transform the code are missing.
+the code transformers required to transform the code are missing.  code
-AST transformers are not needed code transformed ahead of time (loaded
+transformers are not needed code transformed ahead of time (loaded from
-from ``.pyc`` files).
+``.pyc`` files).
 
 
 Rationale
@@ -132,18 +132,18 @@ obfuscator, deobfuscator, and user-assisted decompiler.
 Use Cases
 =========
 
-This section give examples of use cases explaining when and how AST
+This section give examples of use cases explaining when and how code
 transformers will be used.
 
 Interactive interpreter
 -----------------------
 
-It will be possible to use AST transformers with the interactive
+It will be possible to use code transformers with the interactive
 interpreter which is popular in Python and commonly used to demonstrate
 Python.
 
 The code is transformed at runtime and so the interpreter can be slower
-when expensive AST transformers are used.
+when expensive code transformers are used.
 
 Build a transformed package
 ---------------------------
@@ -153,7 +153,7 @@ It will be possible to build a package of the transformed code.
 A transformer can have a configuration. The configuration is not stored
 in the package.
 
-All ``.pyc`` files of the package must be transformed with the same AST
+All ``.pyc`` files of the package must be transformed with the same code
 transformers and the same transformers configuration.
 
 It is possible to build different ``.pyc`` files using different
@@ -181,7 +181,7 @@ If a package does not contain any ``.pyc`` files of the current
 optimizer tag (or some ``.pyc`` files are missing), the ``.pyc`` are
 created during the installation.
 
-AST transformers of the optimizer tag are required. Otherwise, the
+Code transformers of the optimizer tag are required. Otherwise, the
 installation fails with an error.
 
 
@@ -191,10 +191,10 @@ Execute transformed code
 It will be possible to execute transformed code.
 
 Raise an ``ImportError`` exception on import if the ``.pyc`` file of the
-current optimizer tag is missing and the AST transformers required to
+current optimizer tag is missing and the code transformers required to
 transform the code are missing.
 
-The interesting point here is that AST transformers are not needed to
+The interesting point here is that code transformers are not needed to
 execute the transformed code if all required ``.pyc`` files are already
 available.
 
@@ -202,46 +202,57 @@ available.
 Changes
 =======
 
-This PEP proposes to add an API to register AST transformers.
+This PEP proposes to add an API to register code transformers.
 
 The transformation can done ahead of time. It allows to implement
 powerful but expensive transformations.
 
 
-API for AST transformers
+API for code transformers
-------------------------
+-------------------------
 
-Add new functions to register AST transformers:
+Add new functions to register code transformers:
 
-* ``sys.set_ast_transformers(transformers)``: set the list of AST
+* ``sys.set_code_transformers(transformers)``: set the list of code
-  transformers
+  transformers and update ``sys.implementation.optim_tag``
-* ``sys.get_ast_transformers()``: get the list of AST
+* ``sys.get_code_transformers()``: get the list of code
   transformers.
 
-The order of AST transformers matter. Running transformer A and then
+The order of code transformers matter. Running transformer A and then
 transformer B can give a different output than running transformer B an
 then transformer A.
 
-API of an AST transformer:
+API of an code transformer:
 
-* An AST transformer is a callable object with the prototype::
+* An code transformer is a class with ``ast_transformer()`` and/or
+  ``bytecode_transformer()`` methods and a ``name`` attribute. Example::
 
-    def ast_transformer(tree, context):
+    class MyTransformer:
+        name = 'my_transformer'
+
+        def bytecode_transformer(self, bytecode, ...):
             ...
             return tree
 
-  where *tree* is an AST tree and *context* is an object with a
+        def ast_transformer(self, tree, context):
-  ``filename`` attribute (``str``). New attributes may be added to
+            ...
-  *context* in the future.
+            return tree
 
-* It must return an AST tree.
+  ``bytecode_transformer()``: *bytecode* is bytes objects.
-* It must have a ``name`` attribute (``str``): short string used to identify an
+
-  optimizer. The name must not contain ``.`` (dot) nor ``-`` (dash) characters:
+  ``ast_transformer()``: *tree* is an AST tree and *context* is an
-  ``.`` is used to separated fields in a ``.pyc`` filename and ``-`` is used
+  object with a ``filename`` attribute (``str``). New attributes may be
-  to join AST transformer names to build the optimizer tag.
+  added to *context* in the future. It must return an AST tree. It can
-* The transformer is called after the creation of the AST by the parser
+  modify the AST tree in place, or create a new AST tree.
-  and before the compilation to bytecode
+
-* It can modify the AST tree in place, or create a new AST tree.
+* The ``name`` attribute (``str``) must be a short string used to
+  identify an optimizer. The name must not contain ``.`` (dot), ``-``
+  (dash) characters or directory separators: ``.`` is used to separated
+  fields in a ``.pyc`` filename and ``-`` is used to join AST
+  transformer names to build the optimizer tag.
+* The AST transformer is called after the creation of the AST by the
+  parser and before the compilation to bytecode
+* The bytecode transformer is after the compilation to bytecode
 
 .. note::
    It would be nice to pass the fully qualified name of a module in the
@@ -267,23 +278,23 @@ Changes on ``importlib``:
   ``opt``. Remove also the special case for the optimizer level ``0``
   with the default optimizer tag ``'opt'`` to simplify the code.
 * When loading a module, if the ``.pyc`` file is missing but the ``.py``
-  is available, the ``.py`` is only used if AST optimizers have the same
+  is available, the ``.py`` is only used if code optimizers have the
-  optimizer tag than the current tag, otherwise an ``ImportError``
+  same optimizer tag than the current tag, otherwise an ``ImportError``
   exception is raised.
 
 Pseudo-code of a ``use_py()`` function to decide if a ``.py`` file can
 be compiled to import a module::
 
-    def get_ast_optim_tag():
+    def transformers_tag():
-        transformers = sys.get_ast_transformers()
+        transformers = sys.get_code_transformers()
         if not transformers:
             return 'opt'
         return '-'.join(transformer.name for transformer in transformers)
 
     def use_py():
-        return (get_ast_transformers() == sys.implementation.optim_tag)
+        return (transformers_tag() == sys.implementation.optim_tag)
 
-The order of ``sys.get_ast_transformers()`` matter. For example, the
+The order of ``sys.get_code_transformers()`` matter. For example, the
 ``fat`` transformer followed by the ``pythran`` transformer gives the
 optimizer tag ``fat-pythran``.
 
@@ -345,7 +356,7 @@ With the ``'fat'`` optimizer tag:
 
 
 AST transformer
-----------------
+---------------
 
 Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``::
 
@@ -365,13 +376,15 @@ Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``::
         def __init__(self):
             self.transformer = KnightsWhoSayNi()
 
-        def __call__(self, tree, context):
+        def ast_transformer(self, tree, context):
             self.transformer.visit(tree)
             return tree
 
 
-    # register the AST transformer
+    # append our AST transformer after existing transformers
-    sys.set_ast_transformers([ASTTransformer()])
+    transformers = sys.get_code_transformers()
+    transformers.append(ASTTransformer())
+    sys.set_code_transformers(transformers)
 
     # execute code which will be transformed by ast_transformer()
     exec("print('Hello World!')")