PEP 511

* bytecode => code
* more fun with bytecode
* enhance the Changes section

pep-0511.txt

@@ -12,10 +12,11 @@ Python-Version: 3.6
 Abstract
 ========
 
-Propose an API to register AST and bytecode transformers. Add also ``-o
+Propose an API to register bytecode and AST transformers. Add also ``-o
-OPTIM_TAG`` command line option to change ``.pyc`` filenames. Raise an
+OPTIM_TAG`` command line option to change ``.pyc`` filenames, ``-o
-``ImportError`` exception on import if the ``.pyc`` file is missing and
+noopt`` disables the peephole optimizer. Raise an ``ImportError``
-the code transformers required to transform the code are missing.  code
+exception on import if the ``.pyc`` file is missing and the code
+transformers required to transform the code are missing.  code
 transformers are not needed code transformed ahead of time (loaded from
 ``.pyc`` files).
 
@@ -39,16 +40,14 @@ Writing an optimizer or a preprocessor is out of the scope of this PEP.
 Usage 1: AST optimizer
 ----------------------
 
-Python 3.6 optimizes the code using a peephole optimizer. By
-definition, a peephole optimizer has a narrow view of the code and so
-can only implement basic optimizations. The optimizer rewrites the
-bytecode. It is difficult to enhance it, because it written in C.
-
 Transforming an Abstract Syntax Tree (AST) is a convenient
 way to implement an optimizer. It's easier to work on the AST than
 working on the bytecode, AST contains more information and is more high
 level.
 
+Since the optimization can done ahead of time, complex but slow
+optimizations can be implemented.
+
 Example of optimizations which can be implemented with an AST optimizer:
 
 * `Copy propagation
@@ -101,7 +100,9 @@ Usage 2: Preprocessor
 ---------------------
 
 A preprocessor can be easily implemented with an AST transformer. A
-preprocessor has various and different usages. Examples:
+preprocessor has various and different usages.
+
+Some examples:
 
 * Remove debug code like assertions and logs to make the code faster to run
   it for production.
@@ -125,10 +126,49 @@ preprocessor has various and different usages. Examples:
 `MacroPy <https://github.com/lihaoyi/macropy>`_ has a long list of
 examples and use cases.
 
+This PEP does not add any new code transformer. Using a code transformer
+will require an external module and to register it manually.
+
 See also `PyXfuscator <https://bitbucket.org/namn/pyxfuscator>`_: Python
 obfuscator, deobfuscator, and user-assisted decompiler.
 
 
+Usage 3: Disable all optimization
+---------------------------------
+
+Ned Batchelder asked to add an option to disable the peephole optimizer
+because it makes code coverage more difficult to implement. See the
+discussion on the python-ideas mailing list: `Disable all peephole
+optimizations
+<https://mail.python.org/pipermail/python-ideas/2014-May/027893.html>`_.
+
+This PEP adds a new ``-o noopt`` command line option to disable the
+peephole optimizer. In Python, it's as easy as::
+
+    sys.set_code_transformers([])
+
+It will fix the `Issue #2506 <https://bugs.python.org/issue2506>`_: Add
+mechanism to disable optimizations.
+
+
+Usage 4: Write new bytecode optimizers in Python
+------------------------------------------------
+
+Python 3.6 optimizes the code using a peephole optimizer. By
+definition, a peephole optimizer has a narrow view of the code and so
+can only implement basic optimizations. The optimizer rewrites the
+bytecode. It is difficult to enhance it, because it written in C.
+
+With this PEP, it becomes possible to implement a new bytecode optimizer
+in pure Python and experiment new optimizations.
+
+Some optimizations are easier to implement on the AST like constant
+folding, but optimizations on the bytecode are still useful. For
+example, when the AST is compiled to bytecode, useless jumps can be
+emited because the compiler is naive and does not try to optimize
+anything.
+
+
 Use Cases
 =========
 
@@ -199,17 +239,85 @@ execute the transformed code if all required ``.pyc`` files are already
 available.
 
 
+Code transformer API
+====================
+
+A code transformer is a class with ``ast_transformer()`` and/or
+``code_transformer()`` methods (API described below) and a ``name``
+attribute.
+
+For efficiency, do not define a ``code_transformer()`` or
+``ast_transformer()`` method if it does nothing.
+
+The ``name`` attribute (``str``) must be a short string used to identify
+an optimizer. It is used to build a ``.pyc`` filename. The name must not
+contain dots (``'.'``), dashes (``'-'``) or directory separators: dots
+are used to separated fields in a ``.pyc`` filename and dashes areused
+to join code transformer names to build the optimizer tag.
+
+.. note::
+   It would be nice to pass the fully qualified name of a module in the
+   *context* when an AST transformer is used to transform a module on
+   import, but it looks like the information is not available in
+   ``PyParser_ASTFromStringObject()``.
+
+
+code_transformer()
+------------------
+
+Prototype::
+
+    def code_transformer(code, consts, names, lnotab, context):
+        ...
+        return (code, consts, names, lnotab)
+
+Parameters:
+
+* *code*: the bytecode (``bytes``)
+* *consts*: a sequence of constants
+* *names*: tuple of variable names
+* *lnotab*: table mapping instruction offsets to line numbers (``bytes``)
+
+The code transformer is run after the compilation to bytecode
+
+
+ast_transformer()
+------------------
+
+Prototype::
+
+    def ast_transformer(tree, context):
+        ...
+        return tree
+
+Parameters:
+
+* *tree*: an AST tree
+* *context*: an object with a ``filename`` attribute (``str``)
+
+It must return an AST tree. It can modify the AST tree in place, or
+create a new AST tree.
+
+The AST transformer is called after the creation of the AST by the
+parser and before the compilation to bytecode. New attributes may be
+added to *context* in the future.
+
+
 Changes
 =======
 
-This PEP proposes to add an API to register code transformers.
+In short, add:
 
-The transformation can done ahead of time. It allows to implement
+* ``-o OPTIM_TAG`` command line option
-powerful but expensive transformations.
+* ``ast.Constant``
+* ``ast.PyCF_TRANSFORMED_AST``
+* ``sys.get_code_transformers()``
+* ``sys.implementation.optim_tag``
+* ``sys.set_code_transformers(transformers)``
 
 
-API for code transformers
+API to get/set code transformers
--------------------------
+--------------------------------
 
 Add new functions to register code transformers:
 
@@ -222,43 +330,15 @@ The order of code transformers matter. Running transformer A and then
 transformer B can give a different output than running transformer B an
 then transformer A.
 
-API of an code transformer:
+Example to prepend a new code transformer::
 
-* An code transformer is a class with ``ast_transformer()`` and/or
+    transformers = sys.get_code_transformers()
-  ``bytecode_transformer()`` methods and a ``name`` attribute. Example::
+    transformers.insert(0, new_cool_transformer)
+    sys.set_code_transformers(transformers)
 
-    class MyTransformer:
+All AST tranformers are run sequentially (ex: the second transformer
-        name = 'my_transformer'
+gets the input of the first transformer), and then all bytecode
-
+transformers are run sequentially.
-        def bytecode_transformer(self, bytecode, ...):
-            ...
-            return tree
-
-        def ast_transformer(self, tree, context):
-            ...
-            return tree
-
-  ``bytecode_transformer()``: *bytecode* is bytes objects.
-
-  ``ast_transformer()``: *tree* is an AST tree and *context* is an
-  object with a ``filename`` attribute (``str``). New attributes may be
-  added to *context* in the future. It must return an AST tree. It can
-  modify the AST tree in place, or create a new AST tree.
-
-* The ``name`` attribute (``str``) must be a short string used to
-  identify an optimizer. The name must not contain ``.`` (dot), ``-``
-  (dash) characters or directory separators: ``.`` is used to separated
-  fields in a ``.pyc`` filename and ``-`` is used to join AST
-  transformer names to build the optimizer tag.
-* The AST transformer is called after the creation of the AST by the
-  parser and before the compilation to bytecode
-* The bytecode transformer is after the compilation to bytecode
-
-.. note::
-   It would be nice to pass the fully qualified name of a module in the
-   *context*  when an AST transformer is used to transform a module, but
-   it looks like the information is not available in
-   ``PyParser_ASTFromStringObject()``.
 
 
 Optimizer tag
@@ -269,7 +349,7 @@ Changes:
 * Add ``sys.implementation.optim_tag`` (``str``): optimization tag.
   The default optimization tag is ``'opt'``.
 * Add a new ``-o OPTIM_TAG`` command line option to set
-  ``sys.implementation.optim_tag``
+  ``sys.implementation.optim_tag``.
 
 Changes on ``importlib``:
 
@@ -302,6 +382,19 @@ The behaviour of the ``importlib`` module is unchanged with the default
 optimizer tag (``'opt'``).
 
 
+Peephole optimizer
+------------------
+
+By default, ``sys.implementation.optim_tag`` is ``opt`` and
+``sys.get_code_transformers()`` returns a list of one code transformer:
+the peephole optimizer (optimize the bytecode).
+
+Use ``-o noopt`` to disable the peephole optimizer. In this case, the
+optimizer tag is ``noopt`` and no code transformer is registered.
+
+Using the ``-o opt`` option has not effect.
+
+
 AST enhancements
 ----------------
 
@@ -326,8 +419,8 @@ Enhancements to simplify the implementation of AST transformers:
   node type
 
 
-Example
+Examples
-=======
+========
 
 .pyc filenames
 --------------
@@ -355,10 +448,40 @@ With the ``'fat'`` optimizer tag:
 ===========================   ==================
 
 
+Bytecode transformer
+--------------------
+
+Scary bytecode transformer replacing all strings with
+``"Ni! Ni!  Ni!"``::
+
+    import sys
+
+
+    class BytecodeTransformer:
+        name = "knights_who_say_ni"
+
+        def code_transformer(self, code, consts, names, lnotab, context):
+            consts = ['Ni! Ni! Ni!' if isinstance(const, str) else const
+                      for const in consts]
+            return (code, consts, names, lnotab)
+
+
+    # replace existing code transformers with our bytecode transformer
+    sys.set_code_transformers([BytecodeTransformer()])
+
+    # execute code which will be transformed by ast_transformer()
+    exec("print('Hello World!')")
+
+Output::
+
+    Ni! Ni! Ni!
+
+
 AST transformer
 ---------------
 
-Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``::
+Similary to the bytecode transformer example, the AST transformer also
+replaces all strings with ``"Ni! Ni! Ni!"``::
 
     import ast
     import sys
@@ -381,10 +504,8 @@ Scary AST transformer replacing all strings with ``"Ni! Ni! Ni!"``::
             return tree
 
 
-    # append our AST transformer after existing transformers
+    # replace existing code transformers with the new AST transformer
-    transformers = sys.get_code_transformers()
+    sys.set_code_transformers([ASTTransformer()])
-    transformers.append(ASTTransformer())
-    sys.set_code_transformers(transformers)
 
     # execute code which will be transformed by ast_transformer()
     exec("print('Hello World!')")
@@ -397,8 +518,8 @@ Output::
 Other Python implementations
 ============================
 
-The PEP 511 should be implemented be all Python implementation. The AST
+The PEP 511 should be implemented by all Python implementation, but the
-emited by the parser is not specified.
+bytecode and the AST are not standardized.
 
 By the way, even between minor version of CPython, there are changes on
 the AST API. There are differences, but only minor differences. It is
@@ -432,8 +553,13 @@ implementing various optimizations. Most interesting optimizations break
 the Python semantics since no guard is used to disable optimization if
 something changes.
 
-Issue #17515: `Add sys.setasthook() to allow to use a custom AST
+In 2015, Victor Stinner wrote the `fatoptimizer
-optimizer <https://bugs.python.org/issue17515>`_.
+<http://fatoptimizer.readthedocs.org/>`_ project, an AST optimizer
+specializing functions using guards.
+
+The Issue #17515 `"Add sys.setasthook() to allow to use a custom AST"
+optimizer <https://bugs.python.org/issue17515>`_ was a first attempt of
+API for code transformers, but specific to AST.
 
 
 Python Preprocessors
@@ -448,8 +574,8 @@ Python Preprocessors
   preprocessor directives in Python, like ``#define`` and ``#ifdef``
 
 
-Modify the bytecode
+Bytecode transformers
--------------------
+---------------------
 
 * `codetransformer <https://pypi.python.org/pypi/codetransformer>`_:
   Bytecode transformers for CPython inspired by the ``ast`` module’s
@@ -465,10 +591,6 @@ Modify the bytecode
 See also:
 
 * `BytecodeAssembler <http://pypi.python.org/pypi/BytecodeAssembler>`_
-* `Issue #2506 <https://bugs.python.org/issue2506>`_: Add mechanism to
-  disable optimizations
-* `[Python-ideas] Disable all peephole optimizations
-  <https://mail.python.org/pipermail/python-ideas/2014-May/027893.html>`_
 
 
 Copyright